Measuring the Information (or Digital) Society sounds as if it should be simple. After all, it’s digital! If we’re worried about the volume of data that’s gathered about us every time we click a mouse or tap a smartscreen, surely there must be more than enough to tell us all we need to know about how the Information Society’s evolving.
Not so, and especially not so when it comes to measuring the impact of ICTs on sustainable development. Even more especially not so when it comes to qualitative aspects such as human rights, or indirect impacts such as those on the environment.
Measuring the SDGs
Let’s take a look at the Sustainable Development Goals (SDGs). I’ve been working with other experts lately to identify ICT indicators that can help us to assess what progress we’re making towards their achievement. That’s far from easy.
The starting point, of course, should be sustainable development rather than ICTs, the impact of ICTs rather than their presence, what they are doing for or to (or, indeed, perhaps against) progress towards the targets for health, education, women’s empowerment and other Goals set out in the UN’s 2030 Agenda for Sustainable Development.
Policymakers who’re concerned with ICTs’ potential to facilitate sustainable development need to get beyond assumptions about what’s happening towards knowledge based on evidence, and then to evidence-based insights into impacts and potential impacts.
ICTs and SDGs
The SDGs were agreed in 2015 and set targets, mostly for 2030 though some have dates before then.
ICTs were recognised as development enablers in the 2030 Agenda, which includes the SDGs. Though no specific Goal could be agreed concerning ICTs, they’re referenced in targets for several Goals concerned with other themes.
Since the Agenda was agreed, it’s become clear that ICTs are likely to have more influence on economy, society and culture than was envisaged at the time, so they may be more powerful development enablers. But it’s also become clearer that they can and do have negative as well as positive impacts on developmental goals, not least equality.
We need to measure both positives and negatives. There’ve been too many cases where assumptions based on overenthusiasm and inadequate or faulty evidence have led to poorly designed development policies and programmes - to bad ICT4D, if you prefer.
Three types of target
There are three types of target in the SDGs to which ICTs are relevant.
There’s one target among them that’s specifically concerned with ICTs. Goal 9 Target c calls on the international community to ‘significantly increase access to information and communications technology and strive to provide universal and affordable access to the Internet in least developed countries by 2020.’
That isn’t simple. To address it, we need to know what ICTs we’re talking about; what access means (which is not just about connectivity or individuals; not just any connectivity, but broadband connectivity), what we mean by universality and affordability; what we know of trends that are occurring over time.
We can try to measure this in different ways: the population covered by a mobile network or by broadband; the proportion of households or individuals with a computer, mobile, access to the Internet; the volume of online traffic, the cost of ICTs as a proportion of average income. Which is the best indicator? Just one, or a collage of these? And how good are the data that we have in different countries?
Where ICTs are more important
The second set of SDG targets for which ICTs are relevant consists of those where access to and use of ICTs forms a important – perhaps even essential – part of how those targets can be achieved.
Take Target 10 of SDG 16 as an example: to ‘ensure public access to information and protect fundamental freedoms, in accordance with national legislation and international agreements.’ Communications technologies offer new and increasingly essential ways of facilitating this.
Goal 5 target b, similarly, calls for the international community to ‘enhance the use of enabling technology, in particular information and communications technology, to promote the empowerment of women.’
It’s important to remember, though, that what we need to measure here is not whether ICTs are there, but what people do with them and the impacts that result.
Not the extent to which ICTs are available to women and girls, in this case, but the extent to which access and use of them promotes women’s empowerment. Not just gender digital divides, therefore but the differential uses made of ICTs by women and men and the differential outcomes that result.
Where ICTs are less important
The third type of SDGs in this taxonomy are those where ICTs are relevant but relatively marginal to the targets overall.
Evidence on e-waste, for example, contributes to target 8.4 which is concerned with environmental degradation and target 11.6 which is concerned with waste management in human settlements. It’s just part of the picture on those issues – a relatively small part – but measurement’s worthwhile.
While ICTs themselves may have limited direct impact, data derived using ICTs may well also be useful in identifying how far progress is being made towards targets on agriculture, health and education, employment and infrastructure, peace and security.
Four challenges
To measure things effectively, we need to deal to deal with four big data challenges.
Quantity and quality
The first is to do, quite simply, with the quantity and quality of data. It is difficult to find ICT indicators that work across all countries because the data sets we have for many are still poor. In many countries, international data sets still rely on estimates. And even where they’re present, national data sets vary widely in quality, scope and timeliness.
Most importantly, it’s far easier to measure the supply side of ICTs (connectivity) than the demand side (usage) or their impact. A rounded picture needs all three dimensions.
If we’re consistent in the way we handle data and data deficiencies, then we can make valid comparisons between countries and consider trends over time with at least some confidence. But we need better data which have been gathered more systematically, more consistently, more often, with more detail and more granularity.
Disaggregation
The second challenge concerns disaggregation. We need to know much more about the differences in experience within countries if we are to measure access to ICTs, their use and impact on the SDGs effectively.
Most obviously, for instance, we need more information about the different experiences of women and men. The gender digital divide has rightly become a focus for assessment, but reliable data are still in short supply, especially on impact.
If we’re to develop appropriate policies to maximise the value (and minimise the risks) of ICTs for SDGs, we need to know much more about the different experiences of young and old, those with different income levels and educational attainments, those who live in towns and rural areas, ethnic, social and religious minorities and those with disabilities; who benefits and who does not.
Relevance
The third challenge is concerned with relevance. It’s often tempting to treat access to ICTs as a proxy for their use – for example, to use the presence of ICTs in schools as a measure of educational provision and performance; or agreement on an e-health strategy as evidence of improvements in health outcomes.
This is to be avoided. We should use measures of ICT use to measure ICT use; but we need measures of educational and health outcomes to measure educational and health outcomes.
We need clear evidence of associations between those outcomes and the presence of information technologies before we can assert that there are causal links. And we shouldn’t try to measure targets that concern other sectors using ICT indicators if the link between the indicator and the target is too weak or too uncertain. We should only claim what we can prove.
The pace of change
The final challenge (as so often) concerns the pace of change. It’s commonplace to say that rapid changes in technology and markets outpace our ability to respond to them. They also make them very difficult to measure.
-
Data that are more than three years old are often useless, except for comparing then with now.
-
Definitions of terms like broadband will change substantially, immensely even, during the lifetime of the SDGs.
-
The things we seek to measure now may seem irrelevant by 2030.
-
The things we’ll need to measure then may not yet have been invented.
To measure the relationship between ICTs and SDGs effectively we need benchmarks and indicators that make sense from the start date of the SDGs four years ago, make sense now, and will continue to make sense twelve years ahead. Finding those is very difficult, and goes much deeper than the need for better data does alone.