To what extent does a CV reflect someone’s appropriateness for a job? How much does it cost recruiters to read and select candidates from piles of CVs? How much effort and cost goes into developing the optimal resume?
This year in China, 33,000 graduates applied for 70 places on the French cosmetics company L’Oreal’s graduate recruitment scheme. Rather than submitting their CV, they were asked to answer three simple questions via their smartphones. Then, a Shanghai-based startup called Seedlink used predictive language analytics to match right candidates with L’Oreal’s recruitment criteria. Seedlink’s RCXUE product asked open ended questions such as “”If you had one month and a £4,000 budget to tackle any project your heart desired, what would you do?” The software then analysed the language used in the answers, and compared each candidate’s response to draw up a shortlist.
Prior to using this approach L’Oreal filtered candidates by selecting only from China’s top universities. However, L’Oreal is joining a growing band of top firms who are questioning the value of high academic achievement.
Google, for example, recognise that good grades are useful, but not a good indicator of future performance. According to Laszlo Bock, ‘head of hiring’ at Google, quoted in the NYT – “For every job the No. 1 thing we look for is general cognitive ability …. the ability to process on the fly and to pull together disparate bits of information… The second is leadership — i.e. when faced with a problem do you, at the appropriate time, step in and lead… Another is humility and ownership… Research shows that many graduates from hotshot business schools plateau. “Successful bright people rarely experience failure, and so they don’t learn how to learn from that failure.”
The implications of the use of language analytics in assessment are immense. For example, ATC21s, the 21st Century Skills assessment project at Melbourne used language analysis technology to analyse how well students were performing at collaborative tasks. As the technology and its implementation improves the idea of testing students in written examinations so they can pack their CVs with good grades is becoming rapidly dated.
Mass access to the Internet is a mere 20 years old and during this time Web Services have completely revolutionised how we interact – so how will the Internet transform us over the next 20 years?
This article explains how technologies can be architected to allow learning to flourish in the emerging world of the Internet of Things.
Beyond the “Internet of People”
In 2008, the number of things connected to the Internet exceeded the number of people on Earth – but that is still less than 1% of all the physical things in the world today. Cisco’s Internet Business Solutions Group (IBSG) predicts some 25 billion devices will be connected by 2015, and 50 billion by 2020, whilst IDC estimates machine-to-machine communication to grow to 41% of Internet communication by 2020.
IoT represents a major shift in how IT is being used. The personal computer and the ‘Internet of People’ defined the previous IT era. The Internet of Things will be defined by embedded and ubiquitous technologies such as 3d printing, advanced sensing and energy management.
A powerful illustration of this new world comes from wearable clothing, Tshirt OS from Cutecircuit –
Another is the rapid development and spread of 3d printing –
IoT is surging ahead in areas such as manufacturing, medicine and transportation, but what about education? ‘Smart Cities’ initiatives get plenty of attention, but what about Smart Schooling? What about an ‘Internet of Learning-Things’?
To help answer this question, eight schools in the UK will take part in a $1.2m scheme to find out how “Internet of Things” can enhance learning in science, technology, and geography. Students and teachers will be taught to measure and share data – using new Internet of Things technology – in ways that help make learning fun, link directly to the curriculum, and ultimately inform the design of the next generation of schools.
Whilst new-build schools in developed countries routinely use advanced energy and security management IoT technologies, a more fundamental shift is beginning to happen. There is a clear movement towards a Do It Yourself (DIY) approach to technology in the classroom. A great example of this is the such as the Bigshot digital camera kit – http://www.bigshotcamera.com
A key part of this DIY trend is the increasing use of single-board miniature computers, particularly Arduino and Raspberry Pi. Arduino is a purely embedded system, while Raspberry Pi has both embedded and PC functionalities. Both are designed to teach computer science and electronics, and are optimized for managing control technology – i.e. the world of sensors, motors, displays etc (Things).
Floor Turtle and other technologies from the Constructivist movement have been around even longer. However, Arduino and Raspberry Pi have accelerated the Constructivist approach. To get results from these systems, users have to really understand how technology works, and once children understand the basics, their imaginations and creativity are unleashed. In an age when some ‘children think that cheese grows on plants’ one wonders where they think their consumer electronics come from, so its wonderful to see children becoming increasingly connected to the real world of how things work.
Arduino and Rasberry Pi are surrounded by an extensive and complex ecosystem of devices and code, and one of the most noticeable devices is Makey Makey. Coming from the same stable that gave us Lego Mindstorms and Scratch, MaKey MaKey is a circuit-board with crocodile clips and wires which allow users to turn practically any object into a key from a computer keyboard. For example, a banana could be used for the letter ‘A’, some plastercine for the letter ‘B’, and a coin for the letter ‘C’. Using this simple principal, a staircase can be turned into a piano, or graphite pencil marks on paper could be used as a game controller.
Neither the Arduino or Raspberry Pi are anywhere near as prolific as PCs or Tablets, and they sell at a tiny fraction of the volume of the consumer and business devices that find their way into Education – tens of thousands a month as opposed to millions. However, unlike consumer and business PCs and Tablets, Arduino and Raspberry Pi have been designed specifically for education – so do they point the way forward?
The cost of a complete class set of Raspberry Pis (around $35 each) with Internet browsing, productivity tools, peripherals, sensors and devices would cost about ½ that of the equivalent class set of Tablets or PCs. However, the big drawback with Raspberry Pi is that they require patience and high levels of technical competency for their setup and operation – users need to become familiar with Linux and command-line prompts. At present the support ecosystem for Raspberry Pi is less than optimally organized for mass proliferation.
To get a better look at what the Internet of Things can mean for Education, we need to look beyond the ‘DIY’ world and think about a complete architecture for “Internet of Learning-Things”.
Towards an “Internet of Learning-Things”
Needs should drive the design of an Internet of Learning-Things – not the other way round. As with all questions about technology, the first question we need to ask is ‘why’? What new scenarios should an ‘Internet of Learning-Things’ deliver? Here’s some examples:
In the next 20 years machines will take increasing amounts of decisions. In a world where so much can be sensed or observed, security and privacy take on new meanings and relevance. In a world where systems will be managed increasingly remotely, technocrats will control much more of the world we live in. Its critical, therefore, that children get to understand how this completely new world works, and learn how to build and control it. To achieve this understanding, children need to have the opportunity to build systems that combine computer science with electronics and product design.
Science, Technology and Geography.
The use of sensors, data-logging and basic electronics has long been a part of the UK National curriculum, but with a proliferation of low-cost sensors, devices, drones and kits, its reasonable to expect to see an increase in the increasing use and sophistication in the application of these technologies across the world.
For example, the Parrot AR.Drone2.0 enables students to survey an area using a mobile phone. HD video is shot and stored on a USB memory stick, or relayed directly back to the phone. In one package, Science (e.g. physics of flight); Technology (e.g. OS, networking, control); and Geography (e.g. surveys, observations) can be delivered, in a way that is completely engaging for children of all ages.
The key development in this space is the opportunity for children to learn how to code with Scratch, Python and .NET Gadgeteer offering progressive learning pathway. Scratch even has a way to control the GPIO on Raspberry Pi, enabling students to control a range of devices easily.
Ubiquitous and context-aware learning.
With devices able to talk more easily with other devices, augmented reality should spill out from museums turning everyday features in the environment into learning objects. For example, point your phone at a building and see what was there of historical significance in the past; point it at a plant or animal and get key scientific facts; use a phone to control a drone and receive live images of your local neighborhood. Kiosks offer another platform for AR, and Lego have a powerful illustration that shows the kinds of scenarios that AR offers –
Learning through everyday play
A market research study by Tangull America indicated that the market for toys with embedded IT is growing over 15% annually, and will grow to sales of US $146 billion by 2015. Examples include interactive puppets, girls’ toys that share secrets, and “real playmates” – which measure changes in facial expressions and use AI to respond. There are huge opportunities to embed learning tools into children’s toys.
With a greater spectrum of learning opportunities available, and wider use of project-based learning, the potential for more personalized learning increases.
Devices connecting securely to big (and nano) data, content and SRM systems, can enable more and better e-learning services that dynamically adapt to learner’s needs as they evolve.
“The growth of devices connected to the Internet will give learners access to untold sources of authentic data in an environmentally friendly way. Through their Internet connections on multiple devices, learners will collect these data and work with fellow learners and experts around the world to analyse, interpret and manipulate the information and so contribute in a meaningful way to the development of social and scientific understanding, Learning will become more contextualised, relevant and meaningful as a result.”
Dr Michelle Selinger, Director of Education Practice at Cisco Consulting Services
Anytime anywhere high-stake assessment and exams
Nearly everyone on the planet has sat or will sit an examination or another form of high stakes assessment. Device-level security, built on biometric systems such as facial recognition, offer ways to ensure honesty in exams. As well as local devices, routers could be potentially enabled for exam-standard security in designated ‘Examination Zones’.
Towards an Internet of Learning-Things Architecture
The first technical problem that needs to be solved is that every device on the Internet needs an IP address to communicate with other devices. Currently most Internet traffic runs on IPv4, which allows ‘only’ 4.3bn addresses. The current version – Ipv6 – allows 7.9 x 1028 times more addresses, but IPv6 and IPv4 are not interoperable, so the transition is not going to be immediate and smooth.
The next problem to be solved is the development of protocols for data, network, transport, sessions and applications. A lot of work is underway such as MQTT, a machine-to-machine/Internet of Things connectivity protocol, but as yet there are no real IoT standards – unlike the Internet of People, which uses protocols such as http (for hypertext), and XMPP (for IM, presence and chat).
So, achieving any form of architectural standardization for an Internet of Learning-Things is going to take some time.
However, in the meantime, there are concepts and scenarios that can help. One way to look at IoLT architecture is to split it into functional layers, and map existing technologies and services to those layers:
Internet of Learning Things Scenario
A student has learned something significant and has verified the learning through a series of low stakes e-assessments. The student now wants to get full credit for this learning through an accredited examination board (eg, University of Oceania Certificate of Secondary Education). The student finds an accredited ‘Examination Zone’ – a room or an area set up to written examination standard, and monitored for honesty. The student logs onto the examination system, which verifies the user through device level biometric security, then locks down the device to ensure no access to local resources. The student is presented with the questions and types or handwrites the answers. The device pushes an encrypted version of the student’s answers to an E-Exam-Ready Wi-Fi router, (gateway) which relays the data to servers, which also have device level security to verify the validity and security conditions of the student’s responses. From there, the examination response is assessed and credit given in due course, with an encrypted certificate sent back to the student.
Whilst this may seem far-fetched and problematic, it’s worth taking a few moments to compare the kind of advances that have been made in Internet and mobile finance and medicine. For example, diagnostics in medicine is light years ahead of ‘diagnostics’ in education. In an era when we allow sensors to be implanted in the human body to monitor and improve health in the most precise and targeted way, why do we insist that practically everyone on the planet sits down in silence and recall facts from memory on bits of paper in order to get recognition for what they have learned?
Despite phenomenal progress with e-assessment and e-examination in some countries, a recent incident at Kasetsart University in Thailand illustrates just how far other places have to go. Students there were pictured wearing makeshift paper ‘anti-cheating’ devices.
“We need to be ready for a new pace of change in learning”, says Jim Wynn, Chief Education Officer at Promethean.
“We will depend upon the content to be organized in ways which do not hinder learning and also and I think crucially, content will have to reflect next generation pedagogues and not those that are designed for the technology of pencil and paper”.
Another key point made by Jim is that the ‘Do It Yourself’ approach is not going to work on its own universally. “There has to be a balance between explore-and-find-out and directed learning from a wise head”.
Within formal learning, a major challenge is going to be lack of technical capacity amongst the teaching workforce. In developing countries, where some teachers don’t even know what Facebook is, ‘DIY’ will be a real challenge. Teachers in this new world will need to be a lot more technically skilled than they are now, and that will be a significant challenge.
Another challenge is the inertia in the examinations systems, and the cascade effect that it has on schooling as a whole.
One of the biggest challenges of all, however, is the uneven distribution of Internet Access across the world. Whilst it’s fascinating to talk theoretically about the Internet of Learning-Things in the developed world, what happens to those who are left behind from even the Internet of People?
According to the International Telecommunications Union, 39% of the world is not using the Internet. 31% of the developing world, and 77% of the developed world are using the Internet.
There are several initiatives aimed at attacking this problem from different angles. For example, there is potential for using old analog TV bands – VHF/UHF – to deliver Internet access, whilst Project Loon is about delivering Internet access via high altitude balloons.
The Internet of Learning-Things will require significant amounts of virtual teaming. For example, the UK schools ILT pilot project will be led by DISTANCE, a consortium which includes at least 8 organisations, including 3 universities. Interestingly, DISTANCE plans to create a digital information hub using Xively Cloud Services – a cloud platform that is purpose-built for the Internet of Things.
An Internet of Learning-Things may be a long way off for some. However, in the same way that online content is beginning to be a disruptive force in formal schooling in some parts of the world, a new era of ultra-low cost and increasingly connected devices, sensors, displays, security and control technologies, is surely going to accelerate change in a very positive direction.
This white paper – Assess. Analyse. Intervene. From E-Assessment to Personalised Learning – was written to help Ministries of Education, Local Education Authorities and prospective suppliers understand how to build on E-Assessment and E-Examination to create personalised learning experiences. Taking the three key building blocks of Assessment, Analytics and Intervention, the paper defines a Personalised Learning Platform and its interfaces within a broader schooling ecosystem – the Schooling Enterprise Architecture.
The central proposition to this paper is that using data generated by the growing use of E-Examination and E-Assessment process offers significant value for increasing the effectiveness of the schooling systems.
Schooling system needs to constantly innovate and evolve. This paper sets out a vision for how schooling leaders can make learning even more effective by personalising the learning experience for all school students – without introducing unmanageable complexities.
The implementation of the key recommendations of this paper should deliver the following benefits:
Effective learning – Intervention is about developing virtuous cycles of learning, tailored to individual needs
Deep insights – using deep analytics, new and unpredicted patterns can be found that can help inform decision makers about where to focus investments
Timely intervention – whilst E-Assessment takes essential “rear view mirror” snapshots of learning performance, predictive analytics can be used to constantly steer students in the right direction, maximizing the chances of doing well in assessment and examinations
Three interdependent processes combine to deliver a personalized learning experience:
Ongoing assessment from a range of sources is used to gather data about how individuals and groups of students are learning. This data is analyzed to help target students with tailored learning, and to make decisions that lead to increased effectiveness. Using data, interventions can be set up do deal with issues such as reducing drop-out rates; selecting the most effective ways of improving reading and mathematics; and dealing with risks before they become a problem. Ultimately interventions can be tailored for individuals and groups of students.
Each of these processes are interconnected in multiple ways –
The white paper explores these processes and how they integrate and can be implemented.
E-assessment is becoming an increasingly hot topic, with an increasing number of governments around the world taking their first steps in this area. Whilst e-assessment has alway been an option in Learning Management Systems, formalised testing at national scale is a relatively new phenomenon. This article explores the opportunities, risks and architectures associated with delivering e-assessment at scale.
For the clarity, the term “e-assessment” is used here as the collective noun for electronic delivery of High Stakes and Low Stakes testing, diagnostics and examinations. The term also covers both summative and formative testing.
Norway – with over 800,000 school-age students – was the first country to implement national level e-assessment. As part of a national programme for improving education, and after successful trials in 2009 where students took examinations on their laptops, all the national tests for Reading, English and Math are now digital. A large part of Norway’s exams are also conducted digitally.
Students enrol in the exam at least one week before they sit it. On the day of the exam they are given a user-name and password. The PCs that they take the assessments on are owned by the students but provided by the school, so there is a minimum specification for the hardware and browser (HTML 5). It’s acceptable for students to use materials that they have stored on their Hard Drive or a USB but not to gain help over the internet. Schools are 100% responsible for ensuring compliance with the rules, and E-assessments are monitored by a combination of teachers and software.
Arild Stangeland from The Directorate for Education and Training at the Norway Ministry of Education explains that the Norwegian system breaks down into the following components:
Administration of examinations, registration and results/reports
Electronic national tests and diagnostic tests
Collaboration Solution for preparing exams and tests
E-processes for preparation of materials for exams for the students
Each of these components have a separate technical architecture supported by a large stack of applications written in .NET, Java, and Flash, and maintained by the The Directorate for Education and Training. Several hundred servers are used, and BizTalk Server is at the centre of the architecture to co-ordinate traffic between different systems. A locally produced Learning Management System is used to deliver the assessments.
Another country that has implemented a national level e-examination system is Georgia, in Eastern Europe. Microsoft’s Shota Murtskhvaladze reports that school graduation exams are now delivered through a “Computer Adaptive Testing” (CAT) system. Last year, 50,000 schoolchildren took the school-leaving exams in 8 subjects in 1520 public and private schools within an eight-day-long timeframe. The solution was developed by an agency of the MoE’s National Examinations Center.
There are a number or drivers behind the move towards e-examination:
Cost – the English examination system cost ~ $1bn in 2009. Much of this is tied up in paper-based processes – printing, delivering, collecting and scanning papers.
Flexibility – the potential for going beyond what students can physically write on a paper.
Speed and accuracy – the time from sitting the assessment to getting an accurate the result in front of those who need to know is compressed with e-assessment.
Whilst the benefits of moving entirely to electronic assessment are clear, some countries are using technology to manage individual component parts.
The assessment division of British company RM Education handles a range of tasks for a large number of UK and international examination and assessment boards. They deliver authoring, delivery, marking and results services. For example, the company carries out on-screen marking of scanned paper scripts for the International Baccalaureate.
Since 2009, RM Assessment has been working in partnership with Cambridge Assessment, the University’s international exams group, to enable e-assessment in more than 3,000 test venues across 18 countries.
In 2007, Romanian company SIVECO, worked with the Ministry of Education in Lebanon to develop an Examination Management System to manage and automate the examinations processes. Whilst the examination system remains paper based, the solution automates the examination administration tasks.
In Romania in 2011, SIVECO built a solution to publish the results of National High School exams. The solution produced 30 reports showing the results for 200,000 candidates and had to deal with high peak usage in a small time-frame – just 2 days.
To handle the peaks, SIVECO used Cloud technologies – Windows Azure in particular. In this project the Romanian Ministry of Education gained ample processing power, eliminated downtime, and avoided spending $100,000 for a comparable on-premises infrastructure. Romania is far from alone in experiencing peaks in data generation and process – the whole assessment industry experiences significant peaks in demand and load during one or two months of the year, which makes Cloud technologies an ideal candidate for e-assessment solutions.
Cloud technologies are also being used to support e-assessment in Columbia. There, the Instituto Colombiano para la Evaluación de la Educación (ICFES) administers standardised tests to students and has used Cloud technologies to reduce costs and better manage online queries when scores are posted. ICFES moved to a Windows Azure platform in partnership with Asesoftware, and has cut costs by 80% and provided students a faster and more reliable solution.
Taking this a step further, the New South Wales Department of Education and Culture – the largest School District in the Southern Hemisphere – has moved to a complete cloud based e-assessment system for Year 9 Science Standards diagnostic testing (ESSA tests). Working in partnership with Australian company Janison, 65,000 students were tested last year in a comprehensive diagnostic assessment.
Tests online revealed much more about how students were thinking, enabling the NSW DEC to provide high quality advice on how to improve teaching and learning. There were other benefits too – saving $200,000 on server infrastructure costs, saving printing and distribution costs, and gaining a week on marking time over previous years.
So if it’s that easy to do, why aren’t more countries doing it? The main barrier is risk. An assessment system failing during the critical period is headline news, as is inequity and inaccuracy. Many of these risks, however, are inherent in paper based systems too. There are plenty of examples of the wrong papers being delivered to schools, and papers getting lost on return to the examination centres. Like all mission critical IT systems, the key is to architect the system with risk mitigation as a top priority.
A basic building-block view of an e-assessment system looks like this:
Key functions include:
A simplified Azure enabled workflow looks like this:
Exam/Assessment Board produces and signs-off assessment content collaboratively.
Assessment content is pushed into the Cloud and distributed via a Content Delivery Network
Assessment content is cached at school/exam center level after the first student has viewed a particular resource. As candidates enter the examination centre, they are given a username and password on a card.
Just before the assessment starts, policies are enforced on the candidate’s client computer, and the assessment content is cached either in a dedicated application or on the browser. The candidate’s response data is cached locally and periodically sent to the Cloud via the school level cache.
In the Cloud, the candidate’s data sits in a queue, and is then stored in flat tables.
Encrypted data from the Cloud is sent to a data center for longer-term storage and processing and in relational databases. Once all the candidate’s response data is taken from the Cloud to the data warehouse, and the Cloud application is stopped.
Markers grade the work and ensure leveling and normalisation.
Results are collated, reported and analysed.
Results are passed on to relevant agencies for recognition and certificate distribution.
Security and Equity
It’s crucial that candidates are all able to use devices of the same minimal specification, which makes a straight BYOD policy – where any device is acceptable – a difficult proposition.
Enforcing policies on the client computer is a key component. Until recently, attaining ‘lock-down’ would have required each computer to join a domain. Whilst having a Domain and Active Directory joined client computer has many advantages, there is another approach – a solution developed by FullArmor called GPAnywhere. This allows “portable” policies to be created from Group Policy Objects and be applied to any end point including a Virtual Application. This means that any device running Windows can have an Assessment policy applied to them.
Another approach to delivery being considered by some is VDI. The ability to be able to push a virtual assessment desktop to a device and lock it down is appealing as it is potentially a simpler approach. However, there are continuity of service risks with VDI which have yet to be fully tested.
assessment is in its infancy, but many leading examination and assessment authorities are looking carefully into what’s next in this space.
There are thee key areas where assessment has much greater potential than paper based assessment:
Computerised Adaptive Testing (CAT) is a form of computer-based test that adapts to the examinee’s ability level. Medical students at St George’s, University of London using CAT based e-assessment tools are asked to make decisions along a branched narrative in which information and choices available at a later stage depend on the choices the student made earlier.
ACARA – the Australian Curriculum and Assessment and Reporting Authority – takes this a step further and are talking about how to provide candidates with branched routes through the assessment so they get appropriate recognition for what they have learned. A student who struggles with a question or task can be routed along a less demanding pathway, whilst a more able or better prepared student can be routed along a more demanding pathway – both are able to get the best out of the assessment process. Test-takers also do not waste their time attempting items that are too hard or trivially easy.
The New South Wales DEC were able to exploit interactivity when they ran their science tests online. Being able to use interactivity in an assessment opens up a wide range of testing options – for example, asking candidates to build or construct something, conduct virtual experiments, use haptics to test dexterity, or develop an animated scenario. None of these options are practical in a paper and pencil assessment.
21st Century Skills
Whilst we will see paper-based assessment for a long time yet, the pressure is on to find ways of assessing 21st Century skills such as creativity, problem solving, communication and collaboration. Problem Solving is now part of PISA 2012 framework Also, ATC21 – the 21st Century Skills assessment project – is doing some very interesting work in the area of collaborative assessment – www.youtube.com/atc21s One thing is certain – pencil and paper testing won’t help much in diagnosing and assessing whether students have acquired 21st Century Skills or not, so its reasonable to conclude that assessment has a big future.
E-assessment has come a long way in a very short time and is one of the last main barriers to the wider adoption of ICT in schooling. It’s clear that Cloud technology is changing the game here – not only enabling lower cost of service, but also opening the possibility of global e-assessment, with assessment and Examination Boards being able to offer their services to anyone on the planet. With the advent of better biometrics, and new ways of supervising assessments remotely perhaps the most exciting prospect is the notion of assessments being available at any point in one’s lifetime, not just at specified times in the calendar.
Practically everyone on the planet takes many examinations and assessment over their lifetime, so the prospects of this age-old process being made more fair, accurate, helpful, available and engaging are very exciting indeed.
Arild Stangeland, The Directorate for Education and Training, Norway Ministry of Education
Wayne Houlden, Aaron Wittman, Caroline Thompson and Niels Grootscholten, Janison, Australia
Eric Jamieson, Robert Cordaiy, Joanne Sim, Jim Sturgiss, and Penny Gill, from New South Wales DEC, Australia
Peter Adams, ACARA, Australia
Steve Harrington and Dave Patrick, RM Assessment
Alexandru Cosbuc and Florian Ciolacu, Siveco
Bob Chung, FullArmor
Horng Shya Chua and Puay San Ng, Microsoft Singapore; Bjørnar Hovemoen, Microsoft Norway; Shota Murtskhvaladze, Microsoft Georgia; Teo Milev and Ksenia Filippova, Microsoft Central and Eastern Europe; and Brad Tipp, Corporate HQ.