Terms of references

1. Introduction and Context

Liechtenstein Development Service (hereafter LED) intends to contract expert(s) for carrying out the external evaluation of the project Youth Maker Club (hereafter YMC), which is implemented by the Moldovan Association of ICT Companies (hereafter referred to as ATIC). For the downloadable document, find the link to  

LED has supported the reform of the Vocational Education and Training sector (VET) in Moldova, providing knowledge, technical capacity, and financial support. LED is committed to ensure that everyone enjoys the fundamental right to quality and relevant education so that all youth and adults are equipped with in-demand skills for employment, decent work, entrepreneurship, and lifelong learning.

The present ToR offers background information about the project, evaluation purpose, scope, and objectives, as well as the evaluation questions and methodology. Furthermore, the ToR describes the qualifications and selection criteria for the evaluation team.

About the project YMC Phase I

In 2021, ATIC started the implementation of the first phase of the Youth Maker Club, which was preceded by an inception phase (August 2019 – December 2020). The project aims to contribute to the relevance of vocational skills development programs by providing VET students with non-formal learning opportunities to develop in-demand skills for today’s workforce (i.e., technical, digital, and soft skills). Against the backdrop of digitalization, automation, rapid progress in robotics and artificial intelligence, captured by the concept of Industry 4.0, jobs across all industries change substantially through the addition of new tasks and / or the modification of existing responsibilities. In response, workforce is required to adapt to the changing working methods, work organization, and tools. Teaching new technical knowledge and skills is becoming ever necessary, with a growing need to focus also on ‘softer’ skills, such as communication, digital and media literacies, critical thinking, negotiation, and teamwork”^[1]. VET system can therefore be pivotal in equipping youth and adults with the relevant skills required for employment, decent work, entrepreneurship, and lifelong learning, and in facilitating the transition to more digital, green economies and societies.

The project is premised on the notion that VET system must keep up to date with the current technological developments and offer flexible learning pathways so students could explore technologies, gain hands-on experience, and engage in solving real-life problems. Capitalising upon the Tekwill and Fablab facilities available in various locations in Republic of Moldova, alternative learning spaces “Makerspaces” were set up for VET students in Chișinău, Balti and Cahul. These collaborative workspaces were equipped with a variety of maker equipment from high-tech to non-tech, including 3D printers, laser cutters, CNC machines, soldering irons, etc. An offer of technology related trainings is available: CNC programming, Electronics and Programming, Modelling and 3D Printing, 3D Modelling and Laser Cutting, Drones and autonomous systems, Robotics and Mechatronics. The project builds on the assumption that Makerspaces represent an enabler for the acquisition of critical technical skills as well as soft skills such as communication, digital literacies, problem solving, and innovation skills which are demanded by employers and industries.  

The overall goal of the project is to contribute to the increase of VET graduates’ employment rate due to relevant vocational education and training. This will be achieved through the following outcome: Enhanced tech skills and increased awareness about career paths in Maker/tech industry among YMC participants.

The outputs contributing to the outcome are: (i) Created adequate infrastructure to foster and to promote a community of tech enthusiasts to boost applicability of latest trends in innovation, such as prototyping, product design, etc; (ii) Improved non-formal tech vocational education and training for VET students through YMC’s activities; (iii) Established partnerships between YMC and private sector to enrich the VET non-formal educational offer with evolving technologies.

The private sector is one of the key stakeholders in the project. A special focus lies on their engagement in the design of the skills development programs and in enthusing the community of tech savvy young people.

During the course of the implementation of the YMC project, important achievements were attained. They include:

• Creation and equipping of Makerspaces in 3 regions to promote application of the latest technologies for solving various life challenges;
• Establishing of pool of 16 trained mentors able to guide and provide support to young people from VET institutions;
• Development of curricula and provision of 6 tracks of tech workshops for VET students along with different initiatives (maker camps, makeathons, repair cafes);
• Set up of partnerships with public and private institutions for organizing of various events aimed at provision of career guidance in technology.

An external evaluation of the YMC project is planned to assess whether the project progress is on track and in line with its planned outcomes as well as to provide strategic direction and recommendations for the current phase and importantly, for the next phase of the project. Therefore, LED is seeking an independent team of evaluators to undertake this assignment.

2. Evaluation purpose

The evaluation is a mid-term evaluation. It is commissioned to assess the major achievements of the 24 months out of the total length of 36 months. The evaluation shall be based on the international standards and good practice in the field of development evaluation and on the OECD/DAC evaluation criteria, standards and principles for use. These frame the process of project evaluations in recognized standards for quality and provide practical guidance and tools for their implementation.

The purpose of the external evaluation is to assess the project progress and results up to date against the objectives and indicators of achievement, and to draw conclusions and recommendations for consolidating project results in the current and the next phase.

Therefore, the purpose of the external evaluation is twofold:

• Assess the key achievements of the YMC project (backward-looking);
• Provide recommendations for the adaptation of the current phase and for the design of the next project phase (forward-looking).

Learning is at the centre of this evaluation exercise. The evaluation will serve as an opportunity for fostering learning and improving performance. In this sense, the evaluation shall be anchored in relevant information gathered during exchanges with project stakeholders as well as it shall draw upon literature / best practices and international expertise in the topic of (non-formal) vocational skills development, Makerspaces, transitioning to digital economies. LED and ATIC will take the findings and recommendations into consideration when exploring possibilities for the next project. The corresponding report will be handled as an internal document; thus, its internal character should enable a more open discussion about possibilities and limits.

3. Evaluation scope and questions

The external evaluation is expected to assess the overall performance of the YMC project. This assignment shall address its coherence, relevance, effectiveness, efficiency, and sustainability.

The assignment shall address the following questions:

Coherence

• How well is the YMC project embedded in the overall reform of the VET system?
• To what extent do the objectives of this project complement and synergize with those of other VET projects which have similar activities?
• Which synergies and partnerships should be strengthened to ensure a successful next phase?

Relevance

2.1. To what extent is the YMC project relevant to the current context of VET sector reform? To what extent are the outcomes, as specified in the project document, relevant to the objectives of the Education Strategy 2030?

2.2. What are the key relevant aspects of YMC project that should be strengthened and / or scaled-up during the next phase?

Effectiveness

3.1 To what extent has the project achieved its expected outcome or is likely to be achieved by the end of the phase? What were the major factors influencing the achievement of the outcome or the lack thereof?

3.2 How effective has the project been in providing relevant training and in equipping VET students with in-demand skills? What were the main enablers and barriers to engaging VET students with the training programs?

3.3 How effective does the project design and theory of change contribute to the desired outcome of the project?

3.4 What are the essential elements of the YMC project that should be continued and / or scaled up during the next phase, to amplify its effectiveness?

Efficiency

4.1 To what extent has the intervention delivered its outcome and outputs in a timely and cost-effective way?

Sustainability

• To what extent will the benefits of the intervention continue, or are likely to continue?
• What major factors and processes could amplify the sustainability of the intervention during the next project phase?

4. Methodology

The evaluation team shall propose the methodology, discuss, and agree on it with the donor and the implementing agency. This evaluation should be a learning exercise. Therefore, it is expected that the team will conduct a participatory evaluation that shall involve project implementers, key stakeholders, direct and indirect beneficiaries in the evaluation process.

A mixed method design, combining quantitative and qualitative methods, is highly recommended. Such approach will enable the collection of a variety of information and insights for data triangulation – a strategy that can enhance the reliability and credibility of the evaluation findings. The methodology may include, but not limited to, desk-review, survey, interviews, focus groups and consultations with project team members, beneficiaries, and stakeholders. It is expected that the evaluation team will not only collect relevant data but will also jointly reflect with the stakeholders on the researched topics and emerging findings.

5. Evaluation products (deliverables)

The deliverables of the evaluation consist of the following key evaluation products (all documents shall be written in English):

1. Inception report. Within two weeks after the start of the assignment, the evaluation team shall submit an inception report of maximum 5 pages to detail the proposed methodology, sampling framework, data sources, data collection tools and timeframe.
2. Draft evaluation report. The project team and donor representatives must have the possibility to review the draft evaluation report (in electronic form) to ensure that the evaluation meets the required quality criteria and to clarify any misunderstandings. LED and ATIC must have a minimum of two weeks for giving feedback to the draft.
3. Final evaluation report (max. 60’000 characters, annexes not included). The report shall contain the insights and lessons learnt of the evaluation activities and recommendations for adapting the next phase.
4. Evaluation brief describing the main findings, conclusions, and recommendations. It should not exceed 3 to 4 pages which will be published on ATIC website.

6. Team composition and expertise

The evaluation shall be carried out by international or national consultant(s). In case the consultant is an international expert, s/he should work together with a local consultant.

The consultant(s) should have the following qualifications and competences:

• Proven record in leading evaluations in the context of development projects/programmes, e.g. in the field of vocational skills development, STEM learning, youth employability.
• Proven experience and understanding of the Vocational Education and Training system in Moldova and in other contexts.
• Knowledge of, and experience in applying standard evaluation principles, qualitative and quantitative evaluation methods.
• Ability to draft concise evaluation reports of high quality in English.
• One member of the team should be able to communicate fluently in Romanian and Russian.
• Strong sensitivity and responsiveness to bring forward and deal with gender and social inclusion issues.

7. Evaluation ethics

The evaluation team shall operate in an impartial and unbiased manner and give a balanced presentation of strengths and weaknesses of the project being assessed, with due regard to the wellbeing of those involved in the evaluation, as well as those impacted by its results. All information gained through this evaluation must be handled confidentially and cannot be used for other purposes than for this evaluation. The evaluation team shall respect people’s right to provide information in confidence and make interviewed participants aware of the scope and limits of confidentiality, while ensuring that sensitive information of partners cannot be traced to its source.

8. Implementation arrangements

LED and ATIC will select the evaluator. ATIC will contract the evaluator and provide the project documentation (project document, applications, logframe, M&E plan, progress reports). It is proposed to arrange a briefing during which the evaluation team shall discuss the contractual provisions, implementation arrangements, availability of data and shall agree on additional information to be submitted by ATIC before the evaluation starts.

ATIC will assist the evaluation team in arranging visits with stakeholders. Transport and other logistic arrangements are organized by the evaluation team. Any additional information will be provided by the donors and / or implementing agency upon request.

9. Timeframe for the evaluation process

The estimated duration of the contract is September 2023 to January 2024. In order to be able to make maximum use of the evaluation and its recommendations for the planning of the next phase, the final evaluation report must be submitted not later than 10 January 2024. It is up to the evaluation team to propose a feasible schedule with estimated working days to achieve the deadline. The schedule shall foresee the timelines for briefing, desk review, field mission, debriefing, and submission of deliverables. The time planning shall consider that donor and implementing agency will take up to two weeks to review and comment on the draft evaluation report.

Key milestones:

10. Terms of payment

The available budget for the evaluation costs amounts up to 15’000 EUR. The sum includes the evaluators’ remuneration and all the expenses for the services described above, including, travel, accommodation, communication expenses. Any international individual must include within their cost proposal the 12% tax required by Moldovan taxation law. The sum will be paid in instalments linked to the deliverables being accepted:

50 % after submission of the draft evaluation report; yet, there is room for negotiation

50 % after approval of the final evaluation report.

11. Submission of Applications

Interested consultants (individual or teams) who have the relevant expertise and experience are asked to submit the following documents:

1. A cover letter with a brief description of relevant previous experiences in evaluation of projects in Vocational Education and Training, Skills Development, Youth Employability.
2. Two final reports of comparable evaluations recently carried out.
3. Contact details of 2 references expected to support claims of knowledge, skills, and experience.
4. A Curriculum vitae (CV), including references to licenses, certifications, accreditations, etc.
5. A technical and financial offer. The technical offer shall shortly describe the evaluators’ understanding of what is being evaluated and how each question will be answered. The offer must include a concise description of the methodology, qualitative and quantitative data collection, stakeholder participation approach, a draft schedule of activities and deliverables. The financial offer shall contain the estimated costs necessary for the evaluation.

Applications with the above details shall be sent by 12 September 2023 (12 p.m.) to o.poverjuc@led.md with the subject “YMC External Evaluation”. The results from the selection process will be communicated by 18 September 2023.

Proposals will be evaluated and scored by a Commission against the following selection criteria:

• Proposed methodology (relevance, logic, rigor, practicality, etc.) (40%)
• Relevant qualifications and experience of the evaluator(s) (40%)
• Adequacy of the schedule and work allocation with the technical specifications (20%)

^[1] The Digitization of TVET and Skills Systems, ILO, 2020 link