E-assessment – the high stakes strategy

E-assessment is becoming an increasingly hot topic, with an increasing number of governments around the world taking their first steps in this area. Whilst e-assessment has alway been an option in Learning Management Systems, formalised testing at national scale is a relatively new phenomenon. This article explores the opportunities, risks and architectures associated with delivering e-assessment at scale.

For the clarity, the term “e-assessment” is used here as the collective noun for electronic delivery of High Stakes and Low Stakes testing, diagnostics and examinations. The term also covers both summative and formative testing.

Norway – with over 800,000 school-age students – was the first country to implement national level e-assessment. As part of a national programme for improving education, and after successful trials in 2009 where students took examinations on their laptops, all the national tests for Reading, English and Math are now digital. A large part of Norway’s exams are also conducted digitally.

Log-on screen for students taking national e-assessments in Norway

Students enrol in the exam at least one week before they sit it. On the day of the exam they are given a user-name and password. The PCs that they take the assessments on are owned by the students but provided by the school, so there is a minimum specification for the hardware and browser (HTML 5). It’s acceptable for students to use materials that they have stored on their Hard Drive or a USB but not to gain help over the internet. Schools are 100% responsible for ensuring compliance with the rules, and E-assessments are monitored by a combination of teachers and software.

Arild Stangeland from The Directorate for Education and Training at the Norway Ministry of Education explains that the Norwegian system breaks down into the following components:

  • Administration of examinations, registration and results/reports
  • Electronic national tests and diagnostic tests
  • E-examinations
  • Collaboration Solution for preparing exams and tests
  • E-processes for preparation of materials for exams for the students

Each of these components have a separate technical architecture supported by a large stack of applications written in .NET, Java, and Flash, and maintained by the The Directorate for Education and Training. Several hundred servers are used, and BizTalk Server is at the centre of the architecture to co-ordinate traffic between different systems. A locally produced Learning Management System is used to deliver the assessments.

E-assessments in Georgia

Another country that has implemented a national level e-examination system is Georgia, in Eastern Europe. Microsoft’s Shota Murtskhvaladze reports that school graduation exams are now delivered through a “Computer Adaptive Testing” (CAT) system. Last year, 50,000 schoolchildren took the school-leaving exams in 8 subjects in 1520 public and private schools within an eight-day-long timeframe. The solution was developed by an agency of the MoE’s National Examinations Center.

There are a number or drivers behind the move towards e-examination:

  1. Cost – the English examination system cost ~ $1bn in 2009. Much of this is tied up in paper-based processes – printing, delivering, collecting and scanning papers.
  2. Flexibility – the potential for going beyond what students can physically write on a paper.
  3. Speed and accuracy – the time from sitting the assessment to getting an accurate the result in front of those who need to know is compressed with e-assessment.

Whilst the benefits of moving entirely to electronic assessment are clear, some countries are using technology to manage individual component parts.

The assessment division of British company RM Education handles a range of tasks for a large number of UK and international examination and assessment boards. They deliver authoring, delivery, marking and results services. For example, the company carries out on-screen marking of scanned paper scripts for the International Baccalaureate.

RM Assessement have a range of service offerings

Since 2009, RM Assessment has been working in partnership with Cambridge Assessment, the University’s international exams group, to enable e-assessment in more than 3,000 test venues across 18 countries.

In 2007, Romanian company SIVECO, worked with the Ministry of Education in Lebanon to develop an Examination Management System to manage and automate the examinations processes. Whilst the examination system remains paper based, the solution automates the examination administration tasks.

In Romania in 2011, SIVECO built a solution to publish the results of National High School exams. The solution produced 30 reports showing the results for 200,000 candidates and had to deal with high peak usage in a small time-frame – just 2 days.

To handle the peaks, SIVECO used Cloud technologies – Windows Azure in particular.  In this project the Romanian Ministry of Education gained ample processing power, eliminated downtime, and avoided spending $100,000 for a comparable on-premises infrastructure. Romania is far from alone in experiencing peaks in data generation and process – the whole assessment industry experiences significant peaks in demand and load during one or two months of the year, which makes Cloud technologies an ideal candidate for e-assessment solutions.

SIVECO used Azure to handle peak data loads

Cloud technologies are also being used to support e-assessment in Columbia. There, the Instituto Colombiano para la Evaluación de la Educación (ICFES) administers standardised tests to students and has used Cloud technologies to reduce costs and better manage online queries when scores are posted. ICFES moved to a Windows Azure platform in partnership with Asesoftware, and has cut costs by 80% and provided students a faster and more reliable solution.

Taking this a step further, the New South Wales Department of Education and Culture – the largest School District in the Southern Hemisphere – has moved to a complete cloud based e-assessment system for Year 9 Science Standards diagnostic testing (ESSA tests). Working in partnership with Australian company Janison, 65,000 students were tested last year in a comprehensive diagnostic assessment.

Part of New South Wales’s ESSA tests – c/o NSW DEC

Tests online revealed much more about how students were thinking, enabling the NSW DEC to provide high quality advice on how to improve teaching and learning. There were other benefits too – saving $200,000 on server infrastructure costs, saving printing and distribution costs, and gaining a week on marking time over previous years.

Risks

So if it’s that easy to do, why aren’t more countries doing it? The main barrier is risk. An assessment system failing during the critical period is headline news, as is inequity and inaccuracy. Many of these risks, however, are inherent in paper based systems too. There are plenty of examples of the wrong papers being delivered to schools, and papers getting lost on return to the examination centres. Like all mission critical IT systems, the key is to architect the system with risk mitigation as a top priority.

Architectures

A basic building-block view of an e-assessment system looks like this:

Key functions include:

A simplified Azure enabled workflow looks like this:

Using Azure as a key component in delivering e-assessment at scale. This is the kind of approach used by Janison for the NSW DEC ESSEA assessments.
  1. Exam/Assessment Board produces and signs-off assessment content collaboratively.
  2. Assessment content is pushed into the Cloud and distributed via a Content Delivery Network
  3. Assessment content is cached at school/exam center level after the first student has viewed a particular resource. As candidates enter the examination centre, they are given a username and password on a card.
  4. Just before the assessment starts, policies are enforced on the candidate’s client computer, and the assessment content is cached either in a dedicated application or on the browser. The candidate’s response data is cached locally and periodically sent to the Cloud via the school level cache.
  5. In the Cloud, the candidate’s data sits in a queue, and is then stored in flat tables.
  6. Encrypted data from the Cloud is sent to a data center for longer-term storage and processing and in relational databases. Once all the candidate’s response data is taken from the Cloud to the data warehouse, and the Cloud application is stopped.
  7. Markers grade the work and ensure leveling and normalisation.
  8. Results are collated, reported and analysed.
  9. Results are passed on to relevant agencies for recognition and certificate distribution.

Security and Equity

It’s crucial that candidates are all able to use devices of the same minimal specification, which makes a straight BYOD policy – where any device is acceptable – a difficult proposition.

Enforcing policies on the client computer is a key component. Until recently, attaining ‘lock-down’ would have required each computer to join a domain. Whilst having a Domain and Active Directory joined client computer has many advantages, there is another approach –  a solution developed by FullArmor called GPAnywhere. This allows “portable” policies to be created from Group Policy Objects and be applied to any end point including a Virtual Application. This means that any device running Windows can have an Assessment policy applied to them.

FullArmor’s GPAnywhere

What next?

Another approach to delivery being considered by some is VDI. The ability to be able to push a virtual assessment desktop to a device and lock it down is appealing as it is potentially a simpler approach. However, there are continuity of service risks with VDI which have yet to be fully tested.

assessment is in its infancy, but many leading examination and assessment authorities are looking carefully into what’s next in this space.

There are thee key areas where assessment has much greater potential than paper based assessment:
CAT

Computerised Adaptive Testing (CAT) is a form of computer-based test that adapts to the examinee’s ability level. Medical students at St George’s, University of London using CAT based e-assessment tools are asked to make decisions along a branched narrative in which information and choices available at a later stage depend on the choices the student made earlier.

ACARA – the Australian Curriculum and Assessment and Reporting Authority – takes this a step further and are talking about how to provide candidates with branched routes through the assessment so they get appropriate recognition for what they have learned. A student who struggles with a question or task can be routed along a less demanding pathway, whilst a more able or better prepared student can be routed along a more demanding pathway – both are able to get the best out of the assessment process. Test-takers also do not waste their time attempting items that are too hard or trivially easy.

Simulations

The New South Wales DEC were able to exploit interactivity when they ran their science tests online. Being able to use interactivity in an assessment opens up a wide range of testing options – for example, asking candidates to build or construct something, conduct virtual experiments, use haptics to test dexterity, or develop an animated scenario. None of these options are practical in a paper and pencil assessment.

21st Century Skills

Whilst we will see paper-based assessment for a long time yet, the pressure is on to find ways of assessing 21st Century skills such as creativity, problem solving, communication and collaboration. Problem Solving is now part of PISA 2012 framework Also, ATC21 – the 21st Century Skills assessment project – is doing some very interesting work in the area of collaborative assessment – www.youtube.com/atc21s One thing is certain – pencil and paper testing won’t help much in diagnosing and assessing whether students have acquired 21st Century Skills or not, so its reasonable to conclude that assessment has a big future.

Conclusion

E-assessment has come a long way in a very short time and is one of the last main barriers to the wider adoption of ICT in schooling. It’s clear that Cloud technology is changing the game here – not only enabling lower cost of service, but also opening the possibility of global e-assessment, with assessment and Examination Boards being able to offer their services to anyone on the planet. With the advent of better biometrics, and new ways of supervising assessments remotely perhaps the most exciting prospect is the notion of assessments being available at any point in one’s lifetime, not just at specified times in the calendar.

Practically everyone on the planet takes many examinations and assessment over their lifetime, so the prospects of this age-old process being made more fair, accurate, helpful, available and engaging are very exciting indeed.

Additional Information

New South Wales ESSA (Science diagnostics tests)
Norway
Changing faces of assessment

Azure

http://www.windowsazure.com

Thanks to:

Arild Stangeland, The Directorate for Education and Training, Norway Ministry of Education

Wayne Houlden, Aaron Wittman, Caroline Thompson and Niels Grootscholten, Janison, Australia

Eric Jamieson, Robert Cordaiy, Joanne Sim, Jim Sturgiss, and Penny Gill, from New South Wales DEC, Australia

Peter Adams, ACARA, Australia

Steve Harrington and Dave Patrick, RM Assessment

Alexandru Cosbuc and Florian Ciolacu, Siveco

Bob Chung, FullArmor

Horng Shya Chua and Puay San Ng, Microsoft Singapore; Bjørnar Hovemoen, Microsoft Norway; Shota Murtskhvaladze, Microsoft Georgia; Teo Milev and Ksenia Filippova, Microsoft Central and Eastern Europe; and Brad Tipp, Corporate HQ.

Invitation – Schooling Solutions Community

Thanks to the 80 people from 25 countries that took time out from the BETT Show to spend a morning with us at the Schooling Solutions Workshop.

As Roberta Bento from Planeta Educação said – “its amazing how so many of our problems and opportunties are the same”.

Key themes that emerged from the workshop included:

  • Deployment
  • ROI and effectiveness
  • Elearning and Content
  • ITL Research
  • HTML 5
  • Cloud (Live@edu; Azure; InTune)
  • Security

I’d like to thank Bruce Dixon, Sarah Armstrong, Edgar Ferrer Gil, Fotis Draganidis, Dan Baelum, Kirsten Panton, Walid Mohamed, Thomas Hauser and Dolores Puxbuamer for delivering the event.

Schooling Solutions Workshop, London, January 12, 2012

If you are in London on the 12th January for BETT, come and join us at our  Schooling Solutions Workshop.

Key questions that the workshop will address include:

  • How can standards be raised whilst reducing costs?
  • How can you take advantage of trends such as personalization, BYOD, Cloud and virtualization?
  • What approaches can you take to simplify and improve ICT services?

This workshop will bring you up to speed with the latest worldwide trends in education technology and give you practical methods and approaches that you can use immediately. It will be a mix of formal presentation and round-table discussion with world-class experts and leaders in their fields.

Designed to help decision makers plan more effective, efficient and inspiring systems, the workshop will provide an understanding of the Microsoft technology roadmap, solutions for access, connected communities and analytics, and offer the opportunity to work in groups with experts.

Agenda

Time Session
09.00 Solutions for Schooling
10.00 E-Learning
10.30 Institutional Effectiveness and Efficiency
Round-table – project planning sessions
11.00 Access
Managing large scale access programs
Learning
Using ICT to increase learning outcomes
OperationsUsing data to improve decision making
12.30 Reflection & Networking Lunch
  • Date: Thursday, 12th January, 2012
  • Time: 09:00 – 13:00 followed by lunch
  • Location: Microsoft Offices, Cardinal Place, 100 Victoria Street, London, SW1E 5JL

Confirmed speakers/facilitators include Mike Lloyd, Sarah Armstrong, Matthew Fox, Edgar Ferrer Gil, Fotis Draganidis, and Thomas Hauser .

To book your place, contact your local Microsoft Education representative, message me on Facebook, or drop me an email

School Technology Innovation Centre, Brussels

For the next few months I’m delighted to be able to be based out of the School Technology Innovation Centre (STIC) in Brussels. For anyone with a passion for Education Technology this is simply Nirvana!

If you can get to Brussels, put a visit to the STIC at the top of your agenda.

So what is the Brussels STIC? First of all, it’s an education area in the Microsoft Executive Briefing Centre (EBC) in Brussels – an environment dedicated to showing the full spectrum of Microsoft technologies in real-world contexts. The EBC contains a home environment with the latest in 3d television technology, full spectrum of entertainment, electronic music, and home control software. On another floor is a HP Private Cloud datacentre connected directly to the Microsoft Public Cloud in Dublin – complete with a spectrum of demos set up to show what can be achieved with Cloud computing. A large telepresence enables an “in the same room” experience with locations all over the world.

The STIC itself comprises two learning spaces packed with the latest cutting edge education technology. The STIC is run by Jacques Denies, a highly knowledgeable former teacher who is able to demonstrate a wide spectrum of applying technologies to learning scenarios.

Visitors come to the STIC from all over the world. Most visitors are senior decsion makers – for example, within a two week period the entire leadership team of the Thailand Ministry of Education and a delegation including ministers of Education from 14 African states have visited the STIC for workshops.

Finnish Minister of Education, Ms. Henna Virkkunen
European Educators Forum with EU Commissioner Fiegel and ‘DigiGirlz’

The types of activities that are hosted at the STIC range from 3 day workshops to 3 hour briefings/creative brainstorms, to competitions and award ceremonies. World class conferencing facilities and a huge range of equipment make just about any kind of workshop possible to run there.

Conferencing and workshop facilities

The technologies on display at the STIC are far too numerous to list fully here, but a sample include:

The latest in IWB and creative furniture solutions

IWB and creative furniture solutions

Classroom On Wheels

COW

Communication and Collaboration

Roundtable

Assistive Technologies

Assistive Technologies

To book a visit to the STIC, contact Mike Lloyd mike@edutechassociates.net

Specific technologies on display include:

MultiPoint Server + hub (HP & NComputing)

Full spectrum of creativity and productivity desktop applications including Office 365

Windows Phone solutions

SharePoint based e-Learning environment

Assistive technologies on display include:

Ease of Access Center; ZoomIT; Kurzweil; Super Nova; Reporter; Tobii

Mouse alternatives; Specialised keyboards with trigger buttons

Partners include:

Samsung (MyPC, printers, displays)

SMART Technologies (Interactive Whiteboards – tablet)

Sanako (Classroom Management – IT-Lab)

ELearningforce Sharepoint LMS – managed learning

Neo (Mobile Class – Presenter)

OnWijs (Multi-touch applications)

ICTwijs (Assistive Technology)

Sensotec (Assistive Technology)

The “Strategic” Phase

This is the third in a series of articles that aim to help schooling systems develop their technology, the first being “Taking the First Steps“ and the second, Taking the Next Steps – The ‘Enhanced’ Phase.

There are four distinct phases through which technology in schooling evolves. The first phase is characterized by access. In the next phase, technology is used to enhance existing processes. The third phase is characterized by using technology strategically. No longer is technology considered a “bolt-on”, or “veneer” on top of existing processes – it now helps drive schooling towards strategic goals such as significantly improved learning and better return on investments. In the final phase, leading edge schools use ICT to transform their operations, using it to personalize learning, integrate deeply with the wider community, run extremely efficient administration systems and develop a culture of performance.

‘Strategic’ Phase Vision

In the Strategic phase, technology becomes a key asset in achieving the strategic goals of an organisation. It’s about restructuring work and processes and doing things differently.

Typically in this phase, the strategic goals of an organisation would include raising standards and improving performance, and technology is a strategic tool for achieving these strategic goals by enabling:

  • Intelligent intervention – data driven support for learners
  • Connected Learning Communities – fully exploiting all available resources, and integration with the local community
  • Monitoring, analysis and planning – data driven decision making
Four phases of ICT implementation

Goals

Intelligent Intervention

This is essentially about using data to make well informed decisions about what students need to learn or do next. To fully personalise the learning experience students should be constantly assessed as they move through their schooling, and their learning pathways should continuously evolve. This relies on highly effective feedback loops and systems which dynamically adapts to the twists and turns of the learning process, and sets challenging learning goals and tasks. This is extremely difficult to do within a paper-based setup, but can be made a lot easier through using IT systems that provide analytic and workflow capabilities.  Intelligent tutoring systems, and managed learning environments, are becoming more commonplace and increasingly sophisticated.

Monitoring, Analysis and Planning

To manage an organisation strategically, as opposed to fighting fires, the ability to monitor performance, analyse results and plan for improvements is fundamental. Organisations wanting to manage strategically must have three key capabilities:

Monitoring

This capability provides managers with the ability to know “what is happening” and “what has happened.” Organisations implement dashboards, scorecards, or reports to monitor their performance. These visual applications allow managers to keep an eye on important indicators of their organisation’s health.

Analysis

This capability provides managers with the ability to know what is happening and why. To analyse performance, organisations implement solutions that are often very interactive in nature and allow managers to investigate the root cause of issues they see in their dashboards, scorecards, or reports.

Planning

This capability provides the organisation with the ability to model what should happen. Organisations develop processes and tools to conduct the essential planning, budgeting, and forecasting exercises. These processes allow managers to align groups and individuals around the metrics that drive the organisation—for instance: “what are our examination result targets?” or “what is our spending versus our revenue?”

Connected Learning Communities

Whilst there may be elements of learning that require independent work, learning only really acquires meaning in a social context, and the most immediate and direct social context for schooling is the local community.

ICT can be used to connect together all those who can make a contribution to students’ learning – e.g. local business, community resources (e.g. museums/libraries), parents and 3rd party learning services. It can connect students to inspiring individuals and inspirational speakers; promote debate and engagement between collaborators in face-to-face or virtual groupings; and provide mentoring opportunities. Connecting stakeholders together in a Connected Learning Community has enormous benefits such as engaging parents more deeply in the learning process, speeding-up processes and improving students’ connections with the outside world. The core of a connected learning community is a portal that can be accessed from anywhere.

Scenarios

Student Access

In the Strategic phase, students have continual access to their own learning devices. These devices need to enable a range of learning scenarios (not just content consumption), be rugged, easy to repair and support, manageable on a network.

Devices should be available to students so they can learn anytime anywhere, access content, learning management and communication and collaboration tools via the Connected Learning Community Portal.

Anytime Anywhere Learning = access to devices + learning services

Having access to their own devices enables students to experience a wide range of learning scenarios:

ICT enables a wide range of learning styles

Classroom

Classrooms need to accommodate an increasingly wide range of learning styles, and equipment needs to be laid out in quite different ways according to the demands of each different learning task, for example:

Different learning tasks require different floorplans

BECTA provided the following guidance to UK schools on different classroom layout options:

Pods – separate circular / hexagonal / octagonal benches with workstations
Hexagonal pods
Advantages
  • No corners with 2 computers, so no dead spots that cannot be used
  • No extra space required for 2 pupils to share a computer
  • Can support collaborative work as users working around ‘one pod’
Pods – squares with computers on two sides only
Square pods
Advantages
  • No corners with 2 computers, so no ‘dead spots’ that cannot be used
  • No extra space required for 2 pupils to share a computer
  • Can support collaborative work as users working around ‘one pod’
Bays built along walls
Bays
Advantages
  • Teacher can more or less see all computer screens from the centre of the room
  • Provides opportunity to use the centre of the room for tables enabling work away from the computer, and to gather groups for discussion
  • Cabling and electrical work is cheaper and easier than ‘pod’ designs as along the room edge.

School

In the Strategic phase, IT has become a strategic asset to schools. With the infrastructure optimised in the Enhanced phase, we now turn our focus on workloads delivered by servers.

The following services are core in the Strategic phase:

  • Optimised Infrastructure – including File and Print, Database Services, Directory Services, Security, Device Management, and Data Protection and Recovery
  • MIS – Management Information Systems
  • Portal
  • Unified Communication
  • Virtualisation – centralizing computing tasks to improve scalability and system performance

These, typically, will be delivered through three layers:

  • On-Premises – the school hosts key functions on their own servers
  • Data Centre/Private Cloud – the Local Education Authority (LEA) delivers services to schools from their servers
  • Public Cloud – the school receives services from the LEA, Ministry of Education and private suppliers from Public Cloud Services
School Server Infrastructure

Portal

The Strategic phase is characterised by the Connected Learning Community, the core of which is a portal that can be accessed from anywhere. For it to be effective it needs to be “role based” i.e. present users with information and tools relevant to their role and to them as individuals. In other words a teacher in the community sees the information relevant to all teachers, their fellow subject specialists, and also information specific to their particular group of students, their particular HR information, and their particular teaching content, tasks, calendar, e-mail etc.

A portal should give students, parents, managers, teachers, their own “spaces” and deliver to them the resources that are important individually to them through a single web page.  It aggregates information from diverse systems into one interface with a single sign-on ID – and organisation-wide search capabilities so that users can access relevant information quickly.  Teaching and administration staff can use the portal to distribute information to students based on their enrolment, classes, security group or other membership criteria, while enabling them to personalise the portal content and customise the layout to suit their needs.

A great Portal reference architecture is Twynham School. Twynham is a 1600+ Secondary school in Christchurch UK, built  a powerful collaboration platform – “Learning Gateway” – which allows students, staff and parents to work efficiently; develop independent and inter-dependence in their learning strategies; and support children in achieving their full potential. Twynham School won the BECTA ICT Excellence Award in 2008 for learning Beyond the Classroom and the schools works with over 400 schools internationally to support the development of their Learning Platforms.

Twynham School Portal Navigation bar

Mike Herrity at Twynham has published a detailed e-book explaining how the Learning Gateway is used: http://bit.ly/qJohiL

Microsoft have also published a full architectural guide explaining how Twynham built their Learning Gateway – http://bit.ly/qORAW5

Enabling many of the functions in the portal are 2 sub-systems – Content Management and Unified Communications & Collaboration.

Content Management Systems (CMS)

When ICT is fully implemented, vast amounts of content gets created.  In order to get maximum efficiencies from ICT, this content needs to be organised and managed in a way that means that people don’t replicate one another’s work.

A content management system in a connected learning community helps education institutions organise and facilitate the collaborative creation of documents and other content. They enable the full life cycle of content – from initial creation to delivery to end users.  CMS comprise document and records management, web content management, forms, search, library systems, curriculum frameworks, curriculum systems, curriculum exemplars and resource assemblers.

Unified Communications (UC) & Collaboration

Today it is typical that people will have multiple contact addresses – direct line phone number; mobile phone number;  e-mail; Instant Messenger; home number; personal mobile number; home e-mail, etc. Unified Communications (UC) takes identity and presence and then has all of these other ways of interacting simply connect up to that.

A single integrated identity can simplify how you find and communicate with others.  One integrated desktop application can provide easy access to all the ways users are likely to want to communicate.  Another key advantage to UC is that in using Voice over IP (VOIP) for telephone calls, it has the potential to significantly reduce communication costs.

UC enables students, teachers, parents and other stakeholders to confer and consult in the way that suits their work style by switching seamlessly between videoconferencing, telephone, email and instant messaging.

Also within UC are task and calendaring functions.

Data Driven Decision Making

In a schooling system, data driven decision making is supported by a huge number of information systems.  Any process that involves the creation and transmission of information can be considered an information system – even informal discussions.

The collective term for the information systems in schooling is Management Information Systems (MIS).

Functions Supported by an MIS

The functions that a Management Information System need to support are:

Improving Student Performance Progression Management
Learning Management Intelligent Intervention
Parents Engagement In Learning Better Teaching Decisions
Make Better Management Decisions Monitor, Analyse and Plan
Tactical Decision Making Data Visualisation
Manage Resources More Effectively Planning and budgeting
Financial Control Asset Control
Reporting Accountability and Alignment
Performance and Assessment Data KPIs, Scorecards, Dashboards and Reports
Key Performance Indicators (KPIs) Drive Administrative Efficiencies
Planning Organising
Controlling Co-ordinating
Management Information Systems – Functional Architecture

In this context, an information system really means an organised hierarchy of information sub-systems. Management Information System (MIS) is a term used as a container for all of the electronic information systems within a schooling system.  These systems vary in size, scope and capability, from packages that are implemented in relatively small organisations to cover student records alone, to enterprise-wide solutions that aim to cover most aspects of running large multi-site organisations.

A MIS includes the following sub-systems:

  • Decision Support Systems (DSS)
    • Finance
    • Performance Management
    • HR
    • Student Relationship Management (SRM)
    • Enterprise Resource Planning (ERP)
    • Analytics and Business Intelligence (BI)
    • Timetabling
  • Student Information Systems (SIS)
    • Integrated Student Record
    • Electronic grade book
    • Attendance Management
    • Automated workflows
    • E-Forms
  • Learning Platform
    • Learning Management Systems (LMS)
    • Managed Learning Environment (MLE)
    • Virtual Learning Environments (VLE)
    • Content Management Systems (CMS)

For a full description, see Schooling at the Speed of Thought, Chapter 6, Managing Information.

Local Education Authority

In the Strategic Phase, the goal of service provision at Local Education Authority level is to deliver those services which when aggregated improve in quality and price.

Local Education Authorities can use their scale to negotiate the best prices for content, communication, support services etc. Many of the services requiring the most maintenance and management – e.g. learning services, system management, business intelligence, and administrative tasks such as payroll and HR, are delivered more cost effectively from a centralised point.  Other benefits include the use of greater amounts of data for decision making – an LEA with data from many schools can perceive more patterns than a single school with its limited pool of data.

Many LEA services are delivered through data centres built on top of optimised infrastructures. Increasingly data centres will become Private Clouds – essentially Infrastructure as a Service (IaaS) within the data centre. The large scale and pay-as-you-go economics of Public Clouds aren’t available in typical Private Clouds. However, Private Clouds offer at least some of the scalability and elasticity benefits of Public Cloud but with additional control and customisation. Increasingly many of these services will be also be delivered from Public Clouds.

The services delivered by the LEA can be split into two main categories:

  • Schooling Enterprise Services
    • Monitoring, Analysis and Planning
    • Intelligent Intervention
    • Student Relationship Management
    • Administrative Processes
    • Operations
  • E-Learning Services
Local Education Authority Schooling Enterprise Architecture (SEA)

Ministry of Education

Some of the Schooling Enterprise Services delivered by LEAs to their schools and communities could be provided at National level from the Central Ministry of Education. Services such as strategy, policy, budgets, and curriculum are usually set and delegated at national level.

Computing functions at Ministry of Education level can be grouped into three main categories:

  • Internal departments – Curriculum, Policy, Research etc.
  • Regional Services – Resources and BI
  • National Services – Content (information services) and infrastructure – e.g. national level schooling enterprise internet backbone
Ministry of Education perspective

One of the most important functions at Ministry level is to have a “clear line of sight” of the performance of the schooling system. This enables BI analysis and for resources to then be focussed on the areas where they will have most impact.

Fitts and Aziza (Joey Fitts and Bruno Aziza, 2008) talk of a “line of sight” from strategic to operational to tactical decisions as the discipline that drives aligned execution. “Line of sight” means clear visibility of goals, and progress towards them at executive (strategic), management (operational), employees (tactical) levels.

“Clear line of sight” is about performance metric alignment across organisational layers. This can be thought of as an organisation chart for performance metrics, indicating how the various levels of the organisation’s performance metrics relate to one another. At school level, classroom teacher’s metrics roll up to their Head of Department, which in turn roll up to Deputy Principals, which in turn roll up to the Principal. In turn, and depending on the mode of operations, performance metrics for Principals should roll up to those of Local Authority Directors, which in turn finally roll up to the Ministry of Education.

Clear Line of Sight enables strategic allignment

Technology Building Blocks

Finally, pulling these building blocks together we get the following high level architecture:

Technology Building Blocks for Strategic Phase

Conclusion

Moving from the Enhanced Phase to the Strategic Phase is as much about management as ICT. In this phase, the technology is used a tool for getting better allignement between strategy set at MoE level to exectution at school level. At all levels, there are strategic decisions that ICT can help monitor, analyse, plan and execute.

In the next article in this series, we will explore the final phase – Transformation.

Cloud Watching #4 – Managing Learning Content

In the old days it was simple. Agree a curriculum; approve and distribute the books; get teachers to push the contents into empty minds.

Since then everything has changed, especially:

  • The need for students to learn more effectively
  • Student’s appetite for active rather than passive learning experiences
  • Explosive growth of content and ease of access to it

So what does all this mean for learning content, and how it gets managed? On the one hand it could mean chaos as schooling systems deal with extreme complexity – infinite permutations of content types, authoring, storage, categorization, search, access, retrieval, and rendering methods. On the other hand, managed properly, it means the right content built or used by the right person at the right time – making learning significantly more effective. The ease with which ideas, concepts and knowledge are acquired by learners is a function of the availably of engaging learning content and how it is used, so managing content effectively is critical to improving learning effectiveness.

It’s no longer sufficient to think of learning content as a one-way street terminating in the minds of “empty headed” learners. It’s pretty clear that learning is much more effective when students create content rather than just consume it, and the proliferation of easy-to-use content development tools means that students themselves can produce professional standard learning content.

Given the explosion of web content and ease of access to it, the role of publishers is changing quickly too. Publishers have long been considered bastions of authoritative content, but back in 2005 Nature Magazine concluded that Wikipedia and Encyclopedia Britannica were virtually equal in terms of the accuracy of their scientific articles. The challenge for publishers now is to be authoritative, relevant and engaging – not just providing the answers but the conditions in which learners construct their own answers. Learning content has to become much more interactive, immersive, challenging and fun, and it also has to connect to systems that enable intelligent intervention, manage the learning process, and provide analysis.

Schooling systems are faced with bewildering choices when it comes to architecting Learning Content Management Systems (LMCS), so a good place to start is with some questions about what outcomes should be expected from investments in this space. E.g. how do we:

  • Manage content to ensure that the most effective learning takes place
  • Exploit content creation, management, and consumption technologies
  • Leverage new models of content production
  • Ensure that publishers can maintain profitability and invest in R&D
  • Minimise costs and maximise the “Content Economy”

To help frame this discussion we can look to the work of Microsoft Research and their Higher Education project entitled “Technologies for the Scholarly Communications Lifecycle”. Here they describe six distinct areas for supporting the lifecycle of scholarly content. Adapting this for managing learning content within a Schooling Enterprise Architecture we arrive at the following model:

Figure 1. Learning Content Lifecylce for Schooling Enterprises

But before we go any further, what exactly do we mean by learning content?

WHAT IS LEARNING CONTENT?

At one end of the spectrum there are widely available digital entities from which someone can learn – from sophisticated Silverlight or Flash applications to video clips to plain text. At the other end of the spectrum there are highly structured learning content packages designed to meet specific learning objectives.

A key concept in learning content is the “Learning Object” – a self-contained package, with a clear educational purpose containing –

  • Learning content – digital entities including text, images, sound, video
  • Learning tasks
  • Interface to a workflow system so the next learning task can be appropriately set
  • The means by which to assess what learning has resulted
  • Metadata including – learning objective; prerequisite skills; topic; the “interaction model”; technology requirements; educational level; relationships to other learning objects; rights

Ideally, it should be possible to:

  • Edit a Learning Object so it can be tailored to precise requirements
  • Group it into larger collections of content, including longer course structures

Conveniently, there is a standard for how learning objects should be constructed and used. The Sharable Content Object Reference Model (SCORM) is a standard that defines communications between content learning management systems, and how a learning object should be packaged into a transferable ZIP file. (See below for further details).

Advances in technology are also changing views about what actually counts as content.  For example, it could be argued that threads of dialogue through blogs, wikis and instant messaging are forms of content production.

CREATING LEARNING CONTENT

The old steps-and-stages, linear, age-cohorts and classes-dominated, subject-orientated curriculum is being superseded. Its successor is a “Thinking Curriculum”, based on a search for knowledge, on developing competencies rather than consuming content. The Thinking Curriuculum is information rich, multi-layered, and connected.

With the creation of high quality content relatively easy to accomplish, we have to ask a fundamental question – “who gets to produce learning content?” As explored in “High Performance Schools” a key way to get effective learning is to get students to create their own content then get peers to review it. With cheap webcams; basic video editing software; drawing, graphics, and productivity software; web development and portal tools, its increasingly easy to get great results from this approach.

There will always be a role for professionally produced, authoritative content. However, the world of publishing needs to embrace the idea that students and teachers will increasingly want to build their own learning resources from individual learning objects, in much the same way as building models using Lego®.

MANAGING CONTENT

There are essentially two types of content – structured and unstructured. Structured content is that which has been classified, and stored in a way that makes it easy to be found and used. Unstructured content is all other content.

Imposing structure and order on the exponentially expanding unstructured world of user-generated content is a major challenge for all organizations.

 

Figure 2. Unstructured content grows exponentially

Key concepts in Content Management include:

  • Document Management
  • Web Content Management
  • Rich Media Management
  • Archiving and Library Services
  • Scanning (Image and Capture)
  • Document Output Management
  • Workflows
  • Learning Process Management

Learning Content Management Systems (LCMS) help schooling systems organise and facilitate the collaborative creation of learning content, providing developers, authors and subject matter experts the means to create and use learning content. They enable the management of the full life cycle of content – from initial creation to consumption and re-creation by end users. They feature repositories, library systems, curriculum frameworks, curriculum systems, curriculum exemplars and resource assemblers.

A LCMS enables:

  • Efficient search and retrieval
  • Ease of authoring across a learning community
  • Rapid customisation for various audiences

An LCMS should enable seamless collaboration between subject matter experts, designers, teachers, and learners. It should enable content to be made available through a wide array of output types – such as structured e-learning courses, lesson plans, single learning objects – and output devices such as PC, phone or TV.

Learning Content Management Systems differ significantly from Learning Management Systems (LMS) in as much as an LCMS should be used to “feed” content to one or more LMS.

Figure 3. LCMS feeds learning content to LMS

Key LCMS Functions

Based on the Association of Information and Image Management’s specifications, a Learning Content Management System should have the following features and functions:

Categorization/Taxonomy

A taxonomy provides a formal structure for information, based on the specific needs of a schooling system. Categorization tools automate the placement of content (learning objects, documents, images, email, text etc) for future retrieval based on the taxonomy. A key question is who is responsible for and allowed to categorise content, and edit the categorisation data?

Indexing

Additional meta-data supporting information retrieval – this can be based on keywords or full-text.

Document Management

Document management technology helps organisations better manage the creation, revision, approval, and consumption of documents used in the learning process. It provides key features such as library services, document profiling, searching, check-in, check-out, version control, revision history, and document security.

Web Content Management

This addresses the content creation, review, approval, and publishing processes of Web-based content. Key features include creation and authoring tools, input and presentation template design and management, content re-use management, and publishing capabilities.

Digital Asset Management (DAM)

Similar in functionality to document management, DAM is focused on the storage, tracking, and use of rich media documents (video, logos, images, etc.). Digital assets typically have high intellectual property (IP) value.

Repositories

A repository can be a sophisticated system that costs hundreds of thousands of dollars, or a simple file folder system. The key is to have information that can be found once it is placed in the system.

Syndication

Distribution of content for reuse and integration into other content.

Personalization

Based upon data about student learning history, their learning styles and what they next need to learn, types of content and specific learning objects can delivered to best match the student’s needs.

Search/Retrieval

One of the greatest benefits of a well architected LCMS is the ability to get out what you put in with the minimum of effort. Indexing; taxonomy; repository services; relevance; and social cues should make locating specific content in a schooling system easy. Search functions should include:

  • Best Bets
  • Metadata-based Refinement
  • People and Expertise Search
  • Recently Authored Content
  • Defined Scopes
  • Focused Search – site, local, enterprise and web
  • Taxonomy and Term Store Integration
  • View in Browser

Infrastructure Technologies

Supporting these functions are core infrastructure technologies including:

  • Storage
  • Content Integration
  • Migration
  • Backup/Recovery

DRM

Protecting copyrighted content is essential to drive a vibrant “Content Economy”. Ensuring that creators of content get what they deserve for their work is a cornerstone of the Knowledge Economy – the development of which is the aim of many governments. DRM does this by encrypting content to limit usage and copying to limits agreed between the publisher and the customer.

EXPOSING CONTENT

Producing content and storing it is relatively easy, but organizing it to make it easy to find is an altogether different matter. People in large enterprises spend huge amounts of time looking for content, and making it easier to find specific content in schooling systems is core to making them more effective.

Search can help, of course, but the key to making content easy to find is in structuring it well. There is no one right answer for this, but one way of thinking about it is to start by categorising people first and then categorising the content:

Communities

Ideally, content should be exposed to people according to what role they have in the organisation – this is known as “role-based” knowledge architecture. A teacher, for example, should be able to access different content to learners.

Sites

Once communities of users have been defined, sites can be created to serve their specific content needs. Sites are aggregation points for a mix of types of content and methods for surfacing this content.

Libraries

Within a site there can be several libraries, each one categorising content by subject, topic, phase of learning, etc. Categorised content should contain metadata making it easier to find what the user is looking for.

Galleries

For more visual content, it may be easier to flick through a set of images for the user to find what they are looking for – galleries provide this function.

Wikis

A wiki is a website that allows the collaborative creation and editing of interlinked web pages via a browser. This technology has been around for at least 15 years, but its use as a general teaching tool is still in its infancy. However, an increasing number of universities are now adopting them as a teaching tool – see http://www.nytimes.com/2011/05/02/education/02iht-educSide.html?ref=education.

Blogs

Personal spaces for building and publishing content such as blogs or “MySites” give users a way of quickly exposing their thinking to a wider audience to express viewpoints and get feedback.

Figure 4. Structuring content starts with classifying users

LEARNING CONTENT MANAGEMENT ARCHITECTURES

Key Concepts

Roles

A key starting point in architecting a LCMS is determining who the users of the system are and what roles can be assigned to them.

Across the schooling enterprise publishing house staff, experts, teachers, teaching assistants, administrators, students, even parents could all – in theory at least – take on one or more of these roles:

  • Creator – responsible for creating and editing content.
  • Editor – responsible for tuning the content message and the style of delivery, including translation and localisation.
  • Publisher – responsible for releasing the content for use.
  • Administrator – responsible for managing access permissions to folders and files, usually accomplished by assigning access rights to user groups or roles. Admins may also assist and support users in various ways.
  • Consumer, viewer or guest – the person who uses the content after it is published or shared.

Questions raised by the SULINET experience, suggest the following considerations:

  • Who is the principle audience – teachers, students, parents?
  • Who can publish – teachers, students, parents, experts, 3rd party publishers?
  • What incentives are there to encourage contributions?
  • How will Quality Assurance work?
  • What about peer review/rating systems?
  • Should all contributors be allowed to create, publish or edit a Learning Object?
  • Who is the legal owner of a Learning Object – teacher, school, and district?
  • How will logical groupings work? Is it possible/desirable to have national level admin and users, or should groupings work at lower levels such as:
    • District or conglomerate of schools
    • Individual School
    • Grade levels (Eg Year 10)
    • Subject areas (Eg Maths)

Standards

Another key consideration is the role of standards. There are many standards covering content, and the following are the key standards specifically designed for learning content:

SCORM – Sharable Content Object Reference Model – is a collection of standards and specifications for learning objects (Shareable Content Objects, or SCOs). It defines communications between learning objects and a host learning management system. SCORM also defines how content can be packaged into a transferable ZIP file called “Package Interchange Format”. SCORM defines:

  • Content Aggregation Model
  • Runtime Environment
  • Sequencing & Navigation

IMS Global Learning Consortium is concerned with establishing interoperability for learning systems and learning content. IMS publishes specifications for content packaging, enterprise services and digital repositories.

Dublin Core. Defined by the International Organization for Standardization (ISO) The Dublin Core provides metadata descriptions for most learning resources – digital and physical – so they can be described and catalogued. Implementations of Dublin Core typically make use of XML.

CDN

A content delivery network or content distribution network (CDN) caches data at various nodes of a network. A CDN can improve access to the data it caches by increasing access bandwidth and redundancy and reducing access latency. Data content types often cached in CDNs include web objects, downloadable objects, applications, realtime media streams, and database queries.

Blobs

A blob (alternately known as a binary large object, basic large object, BLOB, or BLOb) is a collection of binary data stored as a single entity in a database management system. Blobs are typically images, audio or other multimedia objects, though sometimes binary executable code is stored as a blob.

Scenarios

In the simplest model, the “industrial schooling” approach of pushing book based content into the “empty minds” of learners is digitized:

1. Government sets the curriculum

2. Publishers convert curriculum into content

3. Schools buy content

4. Teacher delivers content

5. Students receive content

Figure 5. Top down approach has limited effectiveness

The SULINET example featured earlier in this blog offers a more sophisticated, “connected learning community” approach. Here, reusable combinations of learning units are stored in a central database. Classification, and the use of metadata and sophisticated enterprise search, makes it easy for users to locate and retrieve content. The smallest digital objects can be independently used or combined together to form learning objects. A curriculum editor application enables users to develop their own learning content.

Extending this further still, in the model below the central repository is connected to external content publishers, online content market places and the worldwide web.  It exploits Cloud technology to drive out infrastructure and management costs; enable flexible scale; and increase reliability and speed.

1. Publishers research and develop new learning packages and make these available for different learning styles

2. Teachers look for materials for specific learning opportunities, and assemble objects into packages for students

3. Teacher assigns learning packages to students

4. Students work in teams to create new content from learning packages

5. Students submits assignment to teacher

6. The best new content from teachers and students gets added to content repository

7. The repository receives content through online market places and the web

8. Standards and processes are overseen by curriculum content committee which uses data to make editorial decisions

Figure 6. An integrated “learning content economy”

Conceptual Design

Converting this usage scenario into a high level conceptual design, we can break down the key processes into three chunks – Creation; Management and Consumption. As discussed at the outset, however Consumption and Creation should increasingly be seen as part of the same process – ie learning is part-consuming and part-producing content.

Figure 7. Conceptual design for a Cloud based Learning Content Management System

Key Products

Creation

Technologies such as Expressions, Visual Studio, and the Adobe Creative Suite are used extensively by professional content developers. DreamSpark is enabling a growing number of students to produce professional quality content too.

Management

Windows and SQL Azure

In the above Schooling Enterprise Architecture Learning Content Management model the core Cloud based content management technologies are Windows and SQL Azure, and the following features are exploited:

  • Compute is a service which runs managed applications in an Internet-scale hosting environment.
  • Storage stores data including blobs – large binary objects, such as videos and images.
  • AppFabric manages users’ permissions and authenticated use of web applications and services, integrated with Active Directory and web based identity systems including Windows Live ID, Google, Yahoo! and Facebook.
  • Content Delivery Network – places copies of web objects (images and scripts), downloadable objects (media files, software, and documents), applications, real time media streams, and other components, close to users. This results, for example, in the smooth streaming of video to Silverlight and Android clients without requiring any software development, management or configuration.

Figure 8. Windows Azure CDN speeds up delivery of content

  • Marketplace – data, imagery, and real-time web services from leading commercial data providers and authoritative public data sources. The Windows Azure Data Marketplace will also contain demographic, environmental, weather and financial datasets. An Application Marketplace will enable developers to easily build applications for Azure.

SQL Azure can also be exploited to provide the following services:

  • Database relational database, providing services to multiple organisations.
  • Data Sync – synchronisation between an organisation’s current SQL on-premises databases and SQL Azure Databases in the Cloud.
  • Reporting – a complete reporting infrastructure that enables users to see reports with visualizations such as maps, charts, gauges, sparklines etc.

Live@Edu

Live@Edu provides a suite of communication, collaboration and storage services for students. It also provides a single account and password for access to many Microsoft Cloud services including Windows Azure. Later this year, Live@Edu will be superseded by Office 365 for Education.

SharePoint Online

SharePoint Online offers a core set of Content Management capabilities including:

  • Document Management
  • Collaboration (team sites), Extranet
  • People Search
  • Content Search
  • Social Computing – including wikis and blogs
  • Publishing Portal (custom theming/branding)
  • Rich Media Management
  • Data Visualization
  • Workflows

 

Figure 9. Through SharePoint, end users get a “control panel” for consuming and creating learning content

Through the SharePoint portal, end users can quickly find the learning content they need, consume and create new content with others, and publish this to a wider connected learning community.

Consumption (and recreation)

Silverlight

Silverlight is a great way for learners to experience learning content. A free, cross-platform browser plug-in, Silverlight is designed for Web, desktop, and mobile applications – online and offline. It supports multimedia, enhanced animation, webcam, microphone, and printing.

Microsoft Learning Content Development System (LCDS)

LCDS is a free tool that enables users to create interactive, online courses and Silverlight learning objects. It can be used to create highly customized content, interactive activities, quizzes, games, assessments, animations, demos, and other multimedia.

Office

PowerPoint is the most widely used content creation tool in schools, and many schools create highly interactive and challenging content with it, eg: see this archive at the University of North Carolina Wilmington

MediaWiki extension for Word allows learning materials developed in Microsoft Office to be saved directly to MediaWiki-based repositories such as WikiEducator.

To create SCORM objects with relatively low levels of technical skill, Hunterstone’s Thesis “Light” is available as a free download with Learning Essentials for integration into Microsoft Office for easy application of the (SCORM) learning content standards to Office documents.

OneNote

Whilst designed as a personal productivity application, OneNote isn’t an Enterprise wide content management solution – however used in the right way, it can be a quick and cost effective way to enable content development, management, search and retrieval amongst small, distributed groups. For example, a teacher could have a “master” OneNote file held on a Windows Live SkyDrive site (in the Cloud). This can contain several “books”, each book sub divided into classes with learning content – videos, links, text etc. Each class can then be further subdivided with an area for each learner. In this way, a Science class – students and teacher, for example, can collaborate with Science classes in other schools.

 

Figure 10. OneNote enables small-scale learning content management

Looking to the Future

HTML 5

The next version of HTML – a language for structuring and presenting content for the World Wide Web – will have profound implications for how learning content can be consumed. It will encourage more interoperable learning content solutions, and will make it easier to include and handle multimedia and graphical content on the web without having to resort to proprietary plugins and APIs.

Conclusion

Providing students with the right kind of learning content at scale is a critical component in making schooling more effective. It’s no longer sufficient to think of content systems as delivery mechanisms, rather they should be thought of as integrated “learning content economies” where learning value is added by all participants and stakeholders. Cloud computing can help facilitate this new approach, driving down costs, increasing connectivity and collaboration, and enabling scalable, flexible and highly available learning content management systems to emerge.

Thanks to David Langridge, Brad Tipp and Sven Reinhardt for support in writing this article.