Chapter 11. Evaluating Digital, Connected and Data-Driven Technologies — Part 1 of 2
11.1 Introduction
Digital technologies increasingly operate as interconnected infrastructures rather than isolated products. The contemporary technological environment is therefore characterized not simply by individual devices, applications or software systems, but by networks of interdependent technologies that exchange information, share computational resources, coordinate activities and influence organizational and social processes.
Cloud computing, Internet of Things systems, digital platforms, big-data systems, digital twins, blockchain, edge computing and connected devices form complex ecosystems in which information moves continuously among people, devices, organizations and computational services. A sensor may collect information from a physical environment, transmit that information through a communication network, process some of the information through an edge device, transfer selected information to a cloud platform, store it in a database, analyze it using artificial intelligence, and return the resulting recommendation to a human operator or automated system. The resulting technological system is therefore substantially more complex than any individual component.
This interconnectedness produces significant benefits but also introduces dependency, cybersecurity, interoperability and governance challenges. A failure in one component may affect other components even when those components are functioning correctly. Similarly, an organization may become dependent on an external cloud provider, communication network, software platform, data provider or specialized technology supplier. The resulting risk is not necessarily visible when each component is evaluated separately.
The evaluation of connected technologies must therefore move beyond component-level performance and examine relationships among components. A technically reliable device may still produce poor overall outcomes if the network through which it communicates is unreliable. A highly accurate analytical system may have limited value if the underlying data are incomplete. A secure application may nevertheless become vulnerable because it depends on an insecure third-party service. A cloud service may provide excellent performance while creating significant organizational dependency through proprietary interfaces and high switching costs.
The concept of technological value must consequently be understood at both the component and ecosystem levels. Evaluation should ask not only whether a technology works, but also whether the wider system remains reliable when technologies interact. This requires examination of interoperability, dependency, data flows, cybersecurity, resilience, governance, lifecycle implications and organizational capability.
Connected technologies also create an important distinction between technological complexity and technological capability. Increasing the number of connected components does not necessarily improve system value. Additional connections can create additional functionality, but they can also create additional attack surfaces, dependencies, maintenance requirements and opportunities for failure. Consequently, technological integration should be evaluated according to the value it creates relative to the additional complexity and risk that it introduces.
Another important consideration is the distribution of control. In traditional technology environments, an organization may own and directly control most of its infrastructure. In connected environments, control can be distributed among cloud providers, telecommunications companies, software vendors, device manufacturers, data processors, platform operators and external service providers. Responsibility can consequently become fragmented. When a failure occurs, determining who is responsible for detecting, reporting, correcting and communicating the problem may become difficult.
Connected digital technologies should therefore be evaluated as socio-technical ecosystems. The technical architecture interacts with organizational structures, human users, business processes, regulatory requirements, contractual arrangements and social expectations. The evaluation framework developed in this book must consequently consider both technological performance and the wider consequences of technological interdependence.
A useful conceptual sequence is:
Physical or digital component → connectivity → data exchange → processing → decision or action → organizational use → stakeholder consequences → monitoring → adaptation.
This sequence illustrates that technological value is generated through relationships rather than through individual components alone. The same technology can produce substantially different outcomes depending on the quality of the surrounding infrastructure, the competence of users, the quality of data, the organizational context and the governance mechanisms established around it.
Connected technology evaluation should therefore consider performance, functionality, reliability, security, privacy, interoperability, scalability, economic value, human consequences, environmental implications, governance and resilience as interconnected dimensions rather than isolated assessment categories.
11.2 Cloud Computing
Cloud computing provides scalable computational resources and can reduce the need for organizations to maintain all infrastructure internally. Instead of purchasing and maintaining extensive physical servers, storage systems and networking infrastructure, organizations can obtain computing resources as services and adjust their capacity according to operational requirements.
The flexibility of cloud computing has significantly changed the economics and architecture of information systems. Organizations can rapidly deploy applications, increase storage capacity, support geographically distributed users and obtain computational resources that might otherwise require substantial capital investment. Cloud environments can also facilitate collaboration, remote working, disaster recovery, data analytics, artificial intelligence and large-scale digital services.
However, cloud computing should not be evaluated solely on the basis of cost savings or scalability. The migration of computing infrastructure to a cloud environment changes the distribution of organizational control. Physical infrastructure may be operated by a third-party provider, data may be stored across multiple locations, applications may depend on proprietary interfaces, and organizational operations may become dependent on continuous network connectivity.
Evaluation should consider:
performance;
availability;
security;
data location;
interoperability;
scalability;
vendor dependency;
exit arrangements;
cost predictability;
regulatory compliance.
Performance evaluation should consider whether cloud resources provide sufficient processing capacity, storage performance, network responsiveness and service reliability for the intended application. Average performance alone may be insufficient because organizations may experience significant performance degradation during periods of high demand. Evaluation should therefore consider peak loads, latency, service-level commitments and performance variability.
Availability is particularly important when cloud services support critical organizational functions. An organization should consider not only the provider's stated availability but also the consequences of service interruption. A short interruption may have relatively minor consequences for a non-critical application but may create substantial financial, operational or safety consequences when cloud infrastructure supports healthcare, financial transactions, industrial systems, transportation or emergency services.
Security must be evaluated across the complete cloud architecture. This includes identity management, authentication, authorization, encryption, network protection, vulnerability management, logging, incident detection and response, backup and recovery. Responsibility for security may be shared between the provider and the customer, and misunderstanding this division of responsibility can create significant vulnerabilities.
Data location is also important because data may be processed or stored in jurisdictions different from the organization's physical location. This may create implications relating to privacy, regulatory requirements, contractual obligations, data sovereignty and cross-border transfers. Evaluation should therefore identify where data are collected, processed, stored, replicated and backed up.
Interoperability is important because organizations rarely operate a single technological system. Cloud services must often communicate with existing databases, applications, enterprise systems, IoT devices and external platforms. Poor interoperability can create duplicated data, manual workarounds, inconsistent information and additional operational risks.
Scalability represents one of the major advantages of cloud computing, but scalability should be evaluated in relation to cost and governance. An organization may be technically capable of expanding computational resources rapidly while simultaneously experiencing unexpected increases in expenditure. Automated scaling can therefore create economic consequences if resource consumption is not monitored.
The economic advantage of cloud services can therefore be offset by long-term dependency if switching costs become excessive. Vendor lock-in may result from proprietary application programming interfaces, specialized databases, unique data formats, contractual arrangements, accumulated organizational knowledge or migration complexity. An organization may technically have the right to leave a provider while practically finding migration prohibitively expensive or disruptive.
Exit arrangements should therefore be evaluated before deployment rather than after dependency has developed. An organization should understand how data can be exported, how applications can be migrated, how service contracts can be terminated, what assistance is available during transition and how long the transition might take.
Cloud evaluation should also distinguish between direct and indirect costs. Direct costs may include computing, storage, networking and licensing charges. Indirect costs may include migration, integration, cybersecurity, staff training, monitoring, compliance, vendor management and future switching costs. Total cost of ownership should therefore be assessed across the expected lifecycle.
A further consideration is concentration risk. If many critical organizational services depend on the same cloud provider or on a small number of providers, a major provider-level disruption may affect numerous organizations simultaneously. This creates a systemic dimension that cannot be understood solely through organizational risk analysis.
Cloud computing should therefore be evaluated through the sequence:
Organizational requirement → cloud architecture → service selection → data and application migration → security controls → operational monitoring → dependency management → resilience → exit capability → lifecycle review.
The central evaluation question is not simply whether cloud computing is technically or economically attractive, but whether the organization can obtain its intended benefits while maintaining sufficient control, resilience, security, interoperability and strategic flexibility.
11.3 Internet of Things
The Internet of Things connects physical objects to digital networks. IoT systems allow physical devices to sense, measure, communicate, process and sometimes act upon information without requiring continuous direct human intervention.
IoT applications include:
agriculture;
manufacturing;
healthcare;
logistics;
transportation;
smart homes;
environmental monitoring.
Gubbi et al. (2013) identified the integration of sensing, communication, cloud computing and data analytics as a major characteristic of IoT ecosystems. Contemporary IoT systems extend this architecture by incorporating edge computing, artificial intelligence, digital platforms, mobile applications, automated control and increasingly sophisticated forms of machine-to-machine communication.
IoT evaluation must therefore consider both physical and digital risks. Unlike conventional software, IoT systems operate at the boundary between digital information and physical environments. A software error in an ordinary information system may produce incorrect information, whereas an error in an industrial controller, medical device, autonomous vehicle or agricultural control system may produce physical consequences.
An IoT system can generally be understood as involving several interconnected layers. Physical objects and sensors generate data. Communication technologies transfer information. Edge or gateway devices may preprocess information. Cloud or centralized systems may provide storage and analytics. Applications transform information into useful outputs, and users or automated systems may subsequently make decisions or initiate actions.
The reliability of the entire IoT system depends on the interaction among these layers. A highly accurate sensor may have limited value if communication is unreliable. A reliable communication network may have limited value if the analytical platform produces incorrect interpretations. An accurate analytical system may nevertheless produce harmful outcomes if automated actions are poorly designed.
Data quality is therefore central to IoT evaluation. Sensor readings may contain noise, missing values, calibration errors, measurement bias or environmental interference. IoT systems should be evaluated according to the accuracy, precision, reliability and contextual appropriateness of the information they generate.
Physical durability is another important consideration. Devices deployed outdoors, in factories, agricultural environments or transportation systems may experience heat, humidity, dust, vibration, corrosion or mechanical stress. Evaluation should therefore consider whether devices remain reliable under the environmental conditions in which they are expected to operate.
Energy consumption is particularly important for battery-powered IoT devices. Frequent communication, continuous sensing and computational processing can reduce battery life. This can increase maintenance requirements and environmental impacts when batteries or devices must be replaced frequently.
Cybersecurity represents another major concern. Every connected device may create an additional entry point into a broader network. Weak passwords, insecure firmware, outdated software, inadequate encryption or poor access control can create vulnerabilities. The evaluation should therefore consider device identity, authentication, firmware security, update mechanisms, network segmentation and monitoring.
Privacy may also become significant because IoT devices can collect information about individuals, households, workplaces and physical environments. Smart home devices, wearable technologies, location systems and connected healthcare devices can produce detailed behavioral information. The evaluation should therefore examine what data are collected, why they are collected, how long they are retained, who can access them and whether secondary uses are permitted.
Interoperability is particularly important because IoT ecosystems may contain devices produced by different manufacturers and operating according to different communication standards. Lack of interoperability can prevent information sharing and create dependence on particular vendors.
IoT evaluation should also consider lifecycle management. Devices may require firmware updates, security patches, battery replacement, calibration, repair and eventual replacement. If devices remain deployed after manufacturer support ends, cybersecurity and reliability risks may increase.
The evaluation of IoT therefore extends from the individual sensor to the wider ecosystem:
Physical device → sensing → communication → edge processing → cloud processing → analytics → decision → physical or organizational action.
The evaluation should examine whether each stage is sufficiently reliable and whether failures at one stage can propagate to other stages.
IoT technologies also illustrate the importance of resilience. If connectivity is interrupted, should the device continue operating locally? If the cloud service becomes unavailable, can critical functions continue? If a sensor provides erroneous information, can the system detect the anomaly before taking action? Such questions distinguish resilient IoT architectures from systems that depend entirely on uninterrupted connectivity.
The growing integration of IoT with artificial intelligence further increases the importance of evaluation. AI systems may interpret sensor data, identify anomalies, predict maintenance requirements or control physical processes. Consequently, the combined system should be evaluated rather than treating the sensor, network and AI model as independent technologies.
The broader principle is that IoT value depends on the reliability of the entire information-to-action chain. Evaluation must therefore consider technical performance, physical conditions, data quality, cybersecurity, privacy, interoperability, maintenance, human interaction, organizational governance and lifecycle sustainability.
11.4 Edge Computing
Edge computing moves processing closer to where data are generated. Instead of transferring all information to a centralized cloud environment for processing, some computation can occur on sensors, gateways, local servers, mobile devices or other computing resources located near the source of data.
Potential benefits include:
reduced latency;
lower bandwidth requirements;
faster local decisions;
improved responsiveness.
These advantages are particularly important in applications where rapid decisions are required. Industrial automation, autonomous vehicles, healthcare monitoring, telecommunications, smart infrastructure and real-time environmental monitoring may benefit from processing information closer to its source.
Reduced latency can be especially important when delays could affect operational outcomes. An industrial control system, for example, may require immediate responses to changes in temperature, pressure or equipment condition. Sending every measurement to a distant cloud service may introduce unnecessary communication delays.
Edge computing can also reduce bandwidth requirements because not every raw data point needs to be transmitted to a central system. Local processing can identify important events, summarize information or filter unnecessary data before transmission. This can be valuable in environments containing thousands or millions of connected devices.
However, distributed computing can create additional security and maintenance challenges. Instead of protecting a relatively centralized computing environment, organizations may need to secure large numbers of distributed devices and local processing environments.
Physical security becomes more important because edge devices may be deployed in locations that are not controlled like traditional data centres. Unauthorized physical access may allow attackers to manipulate devices, extract information or interfere with software and hardware.
Software maintenance can also become more complex. A centralized system may allow an organization to update software in one environment, whereas edge environments may involve numerous devices with different configurations, versions and connectivity conditions. Evaluation should therefore examine whether secure and reliable remote updating is possible.
Edge computing also introduces a potential consistency challenge. Different edge devices may process information locally and temporarily operate with different versions of software, models or data. The evaluation should therefore consider synchronization, version control and configuration management.
The distribution of computation also affects cybersecurity architecture. Security controls must operate across devices, gateways, communication networks, cloud environments and management platforms. A vulnerability in one edge component may provide an attacker with a pathway into a broader system.
Privacy can nevertheless benefit from edge processing when sensitive information can be processed locally rather than transferred unnecessarily to centralized systems. For example, a device may identify an event locally without transmitting all raw sensor information. However, privacy benefits depend on actual system design and should not be assumed simply because edge computing is used.
Edge computing also has environmental and economic implications. Local processing requires computing resources at the edge, including processors, memory, power supplies and cooling. Moving computation from centralized data centres to many distributed devices does not automatically reduce total energy consumption. Evaluation should therefore examine the complete lifecycle and energy profile.
Edge computing can be understood through a distributed architecture:
Data generation → local processing → communication → centralized processing where necessary → decision → action.
The appropriate distribution of computation should depend on latency requirements, data sensitivity, computational requirements, network availability, energy constraints, security and resilience.
A key evaluation question is therefore not whether processing should occur in the cloud or at the edge in absolute terms, but which distribution of computation provides the required combination of performance, security, privacy, resilience, cost and sustainability.
Edge computing also reinforces the importance of hybrid architectures. Many modern systems will use a combination of device-level processing, edge computing and cloud computing. Evaluation should consequently examine how these layers interact rather than evaluating each layer independently.
A useful sequence is:
Data generation → edge processing → local decision capability → secure communication → cloud integration → centralized analytics → monitoring → updating.
The overall value of edge computing depends on whether this distributed architecture improves the intended function without creating disproportionate security, maintenance, governance or lifecycle burdens.
11.5 Digital Twins
Digital twins create digital representations of physical systems. A digital twin may represent a machine, building, factory, transportation system, energy installation, infrastructure asset or other physical entity. Depending on the architecture, the digital representation may receive continuous information from the physical system and may be used to monitor, simulate, predict or optimize its behavior.
Applications include:
manufacturing;
infrastructure;
healthcare;
energy;
transportation.
Tao et al. (2019) emphasize the potential of digital twins to integrate physical and digital systems across the lifecycle of products and processes. The significance of the digital twin therefore extends beyond visualization. A sufficiently integrated digital twin can support design, monitoring, maintenance, simulation, optimization, lifecycle management and decision-making.
Evaluation should examine the accuracy of the relationship between the physical system and its digital representation. A digital model may appear sophisticated while providing limited decision value if the information connecting it to the physical system is incomplete, outdated or inaccurate.
This creates an important distinction between a digital model and a digital twin. A static digital model may describe a physical object or process, whereas a digital twin generally involves a continuing relationship between the physical and digital representations. The strength of this relationship is therefore central to evaluation.
Data synchronization is one of the most important evaluation considerations. If the physical system changes while the digital representation remains unchanged, decisions based on the digital twin may become unreliable. Evaluation should therefore examine update frequency, data latency, synchronization mechanisms and the treatment of missing information.
Sensor quality also affects digital-twin performance. If sensors provide inaccurate measurements, the digital representation may reproduce those errors. The apparent sophistication of simulation or visualization cannot compensate for poor underlying information.
Digital twins can also be used to simulate alternative scenarios. Organizations may examine how a physical asset could respond to different operating conditions, maintenance schedules, environmental conditions or design changes. Such capabilities can support preventive maintenance and planning.
However, simulation results remain dependent on the assumptions embedded in the model. A digital twin should therefore not be interpreted as a perfect representation of reality. Differences between the model and the physical system may arise from incomplete data, simplified assumptions, unexpected environmental conditions or changes in system behavior.
Model validation is therefore essential. Evaluation should consider whether the digital representation has been tested against observed physical outcomes and whether discrepancies are identified and managed.
Digital twins may also incorporate artificial intelligence and predictive analytics. AI can be used to identify patterns, predict equipment failures, optimize resource allocation or detect anomalies. This creates additional requirements for data quality, model validation, explainability, monitoring and governance.
Cybersecurity is particularly important because a digital twin can provide detailed information about physical infrastructure. Unauthorized access could expose operational information or potentially provide pathways toward the physical system itself. Security evaluation should therefore consider the digital twin as part of the wider cyber-physical architecture.
The consequences of incorrect digital-twin decisions may vary substantially by application. An inaccurate digital representation of a non-critical commercial asset may have limited consequences, whereas inaccurate information about a power system, aircraft component, medical device or transportation infrastructure could have serious consequences.
Digital twins also support lifecycle thinking. During design, they can support simulation. During production, they can support quality management. During operation, they can support monitoring and predictive maintenance. During maintenance, they can support diagnosis and planning. At the end of life, they may support decommissioning, reuse or recycling decisions.
A useful lifecycle sequence is:
Physical system → sensing → data transmission → digital representation → model validation → analysis/simulation → decision → physical action → new data → synchronization.
This demonstrates that a digital twin is not a static technological object but a continuously evolving socio-technical system.
Evaluation should therefore consider technical fidelity, data quality, synchronization, model validity, cybersecurity, interoperability, usability, decision consequences, lifecycle value and governance.
The economic evaluation should also examine whether the digital twin creates measurable value. Potential benefits may include reduced downtime, improved maintenance, reduced resource consumption, improved safety and better planning. However, implementation costs may include sensors, communication infrastructure, software, modelling, data storage, cybersecurity, skilled personnel and ongoing maintenance.
The environmental dimension should also be considered. Digital twins may reduce physical experimentation, improve maintenance and optimize resource use, but they require computing infrastructure, sensors, communications and storage. Their environmental value should therefore be assessed across the lifecycle rather than assumed from their digital nature.
A central evaluation question is whether the digital twin provides sufficiently accurate, timely and actionable information to improve decisions about the physical system. The value of the technology therefore lies not merely in creating a digital representation but in creating a reliable and useful relationship between the digital and physical environments.
11.6 Big Data
Big data is frequently characterized by dimensions such as volume, velocity, variety and related properties (Kitchin, 2014). The concept reflects the increasing scale, speed and diversity of data generated by digital platforms, connected devices, transactions, sensors, social systems, scientific research and organizational processes.
However, large amounts of data do not automatically produce good decisions. Data volume is only one characteristic of information quality. A large dataset may contain systematic errors, biased observations, duplicated records, missing information, inappropriate measurements or data that are irrelevant to the question being investigated.
Evaluation should examine:
relevance;
quality;
provenance;
representativeness;
timeliness;
governance;
analytical validity.
Relevance concerns whether the data are appropriate for the purpose for which they are being used. A dataset may be extremely large but still provide little useful information if it does not adequately represent the phenomenon under investigation. The first question should therefore be whether the data are capable of addressing the intended problem.
Quality includes accuracy, completeness, consistency, validity and reliability. Large-scale datasets may contain errors that are difficult to detect because the volume of information makes manual inspection impractical. Automated data-quality procedures may therefore become necessary.
Provenance refers to the origins and history of data. Organizations should understand where data came from, how they were collected, what transformations occurred, who modified them and how they moved through different systems. Without adequate provenance, it may be difficult to establish whether analytical conclusions are trustworthy.
Representativeness is particularly important when big data are used to understand populations or make decisions about individuals. Data generated by digital systems do not necessarily represent the entire population. Individuals with limited digital access may be underrepresented, while highly active users may be overrepresented.
This creates an important distinction between data abundance and population representation. A dataset containing millions of observations can still produce systematically distorted conclusions if the underlying population is poorly represented.
Timeliness is another important consideration. Some applications require real-time or near-real-time information, whereas others may rely on historical information. Data that were accurate when collected may become less useful when circumstances change. Consequently, data evaluation should consider whether information remains appropriate for the decision being made.
Governance determines how data are controlled and used. Big-data environments may involve multiple organizations, platforms, vendors and jurisdictions. Evaluation should therefore examine ownership, access rights, privacy, retention, security, data sharing, ethical use and accountability.
Analytical validity concerns whether the methods used to transform large datasets into conclusions are appropriate. Sophisticated algorithms cannot compensate for an inappropriate research design. Statistical associations may be mistaken for causal relationships, patterns may result from confounding variables, and predictive performance may deteriorate when models are transferred to different environments.
Big-data analysis therefore requires methodological discipline. The evaluation process should distinguish descriptive, diagnostic, predictive and prescriptive uses of data. Describing a pattern does not necessarily explain its cause, and predicting an outcome does not necessarily establish that an intervention will produce the desired result.
Data preprocessing also deserves careful attention. Cleaning, transformation, normalization, feature construction and data integration can substantially influence analytical outcomes. Decisions made during preprocessing should therefore be documented and evaluated.
Big-data systems can also create feedback loops. An analytical system may influence behavior, which subsequently generates new data that are used to retrain or update the system. This can reinforce existing patterns or produce new forms of bias. Continuous monitoring is therefore required.
The relationship between big data and artificial intelligence further increases the importance of evaluation. AI systems depend heavily on the data used to train, validate and operate them. Consequently, weaknesses in data collection or governance can propagate into automated decisions.
Privacy is also a central consideration. Large datasets may allow organizations to combine apparently harmless pieces of information and derive sensitive conclusions. Even when direct identifiers are removed, combinations of attributes may permit re-identification or inference.
The economic value of big data should likewise be evaluated carefully. Data may support improved forecasting, operational efficiency, customer understanding, research and innovation. However, organizations may incur substantial costs associated with storage, processing, integration, cybersecurity, governance and specialist personnel.
Environmental considerations are also relevant because large-scale data storage and processing require energy and physical infrastructure. The growth of data volumes may therefore increase computational and storage requirements. Data retention policies should consequently consider not only organizational value and regulatory requirements but also the cost and environmental implications of retaining unnecessary information.
Big-data evaluation can be represented as:
Data generation → collection → integration → storage → preparation → analysis → interpretation → decision → action → new data → monitoring and reassessment.
The central principle is that data should be evaluated according to their fitness for purpose rather than their quantity alone. More data can improve decision-making when they are relevant, accurate, representative and appropriately analyzed, but additional data can also increase complexity, cost, privacy exposure and opportunities for error.
The evaluation of big-data technologies should therefore examine the entire data lifecycle. This includes how information is generated, collected, transferred, stored, processed, analyzed, shared, interpreted, retained and ultimately disposed of.
Within the broader multidimensional technology evaluation framework, big data should consequently be understood not simply as a technical resource but as an organizational and socio-technical capability. Its value depends on the relationship among data quality, analytical methods, technological infrastructure, human expertise, governance, ethical safeguards and decision-making processes.
The central question is therefore not how much data an organization possesses, but whether the organization can transform appropriate data into reliable knowledge and responsible decisions while controlling the associated technical, economic, social, ethical, privacy, security and environmental risks.
11.7 Data Quality
Data quality is a central technology-evaluation issue.
Poor data can create:
incorrect decisions;
biased models;
unreliable forecasts;
operational inefficiency;
regulatory problems.
A technology evaluation should therefore assess data quality before evaluating the sophistication of analytics.
The importance of data quality becomes particularly significant in interconnected digital environments because data are rarely used only once or by one system. Information generated by one application may be transferred to another system, incorporated into an analytical database, used to train an artificial intelligence model, displayed on a management dashboard and subsequently used to make operational decisions. An error introduced at the beginning of this chain may therefore propagate through several technological and organizational processes.
Data quality should consequently be understood as a multidimensional concept rather than simply the absence of obvious errors. Important dimensions include accuracy, completeness, consistency, validity, timeliness, uniqueness, relevance, reliability, integrity and provenance. The relative importance of these dimensions depends on the intended purpose of the data.
Accuracy concerns whether recorded information correctly represents the phenomenon it is intended to measure. A temperature sensor that consistently reports a temperature several degrees higher than the actual temperature may produce inaccurate information even if its readings are internally consistent. Similarly, a customer database containing incorrect addresses can create operational and communication problems even when the database itself is technically available and well organized.
Completeness concerns whether sufficient information is available for the intended purpose. Missing information may be relatively harmless in one application but critical in another. Missing medical information, missing financial transactions or missing safety information may substantially affect decisions.
Consistency concerns whether information remains compatible across systems and over time. A customer's name, identification number or account status may appear differently in different databases. Such inconsistencies can create duplicate records, incorrect associations and operational errors.
Validity concerns whether data conform to specified rules or expected formats. Dates, identification numbers, transaction amounts and categorical variables may require particular formats. Invalid values can interfere with automated processing and analytical systems.
Timeliness concerns whether data are sufficiently current for the decision being made. Information can be technically accurate but operationally outdated. A traffic-management system, financial-trading platform or emergency-response system may require information within seconds or minutes, whereas historical research may legitimately use information collected years earlier.
Relevance concerns whether the data actually address the problem being evaluated. Organizations sometimes collect large quantities of information without clearly establishing how the information contributes to decision-making. Data accumulation should therefore not be confused with information value.
Uniqueness is also important because duplicate records can distort analysis and operational processes. A single customer appearing multiple times in a database may cause incorrect calculations of customer numbers, transaction volumes or service utilization.
Provenance is particularly important in advanced digital systems. Organizations should be able to identify where data originated, how they were collected, what transformations were performed, which systems processed them and who or what modified them. Provenance provides an evidentiary foundation for evaluating the reliability of subsequent analyses.
Data quality should also be evaluated in relation to context. A dataset may be appropriate for one purpose but inappropriate for another. Information collected for marketing purposes, for example, may not automatically be suitable for medical, legal, educational or safety-critical decision-making.
Historical data require particular attention. Data may accurately describe the past while becoming less representative of current conditions. Changes in consumer behavior, economic conditions, technology, demographics, regulations or environmental circumstances can reduce the usefulness of historical information.
Data quality also has a temporal dimension. The quality of data should therefore be monitored rather than assessed only once. A dataset that meets quality requirements at the time of implementation may deteriorate as systems change, new data sources are introduced and organizational processes evolve.
Human behavior is another source of data-quality problems. Employees, customers and other users may enter incorrect information, misunderstand fields, intentionally manipulate records or use systems in unexpected ways. Data quality should therefore be evaluated as a socio-technical issue involving people, processes and technology.
Automated data validation can improve quality but should not be regarded as a complete solution. Automated checks may identify values that violate predefined rules but may not identify information that is technically valid yet conceptually incorrect. Human review may therefore remain necessary in important applications.
Data quality is especially important in artificial intelligence and machine learning. Poor training data can influence model performance, fairness and generalization. If a dataset systematically excludes certain populations or conditions, the resulting model may perform poorly for those groups even when overall performance appears strong.
The relationship can be represented as:
Data generation → collection → validation → storage → transformation → analysis → decision → outcome → quality monitoring.
This sequence demonstrates that data quality is not a single technical property. It is maintained through a continuing lifecycle.
Organizations should also distinguish between data quality and data quantity. Increasing the amount of poor-quality information does not necessarily improve decision-making. In some circumstances, additional low-quality information may make analysis more difficult by increasing noise, duplication and computational requirements.
Data quality should therefore be evaluated against the intended decision, the consequences of error and the requirements of the technological system. The greater the potential consequence of an incorrect decision, the stronger the evidence required concerning data quality.
Within the multidimensional technology evaluation framework, data quality therefore functions as a foundational condition. Technical sophistication, artificial intelligence, analytics and automation cannot reliably compensate for data that are fundamentally unsuitable for the intended purpose.
11.8 Interoperability
Interoperability refers to the ability of different systems to exchange and use information effectively.
Lack of interoperability can create:
duplicated data;
manual work;
inconsistent records;
higher costs;
vendor dependency.
Interoperability should therefore be treated as both a technical and strategic criterion.
Interoperability becomes increasingly important as organizations move from individual applications toward interconnected digital ecosystems. Modern organizations frequently use numerous databases, cloud services, enterprise applications, IoT devices, communication platforms, analytical systems and external services. These systems must often exchange information even though they may have been developed by different vendors and according to different technical architectures.
Interoperability is more than the ability to transfer data. A system may technically send information to another system while the receiving system is unable to interpret or use that information correctly. Effective interoperability therefore requires compatibility at several levels.
Technical interoperability concerns whether systems can establish communication and exchange information. Syntactic interoperability concerns whether information is structured in a format that the receiving system can interpret. Semantic interoperability concerns whether both systems understand the information in sufficiently similar ways. Organizational interoperability concerns whether the processes, responsibilities and institutional arrangements surrounding the systems allow information to be used effectively.
For example, two healthcare systems may both contain a field called "patient status," but if the two systems use different definitions, the exchange of information may create misleading results. The technical connection may function correctly while the organizational and semantic relationship remains defective.
Interoperability should therefore be evaluated from the perspective of meaning as well as communication.
Standards can play an important role in improving interoperability. Common data formats, communication protocols, metadata structures and interface standards can reduce the difficulty of connecting systems. However, standards themselves do not guarantee effective interoperability. Organizations must also implement them consistently and maintain them as systems evolve.
Interoperability has a direct relationship with organizational efficiency. When systems cannot exchange information automatically, employees may need to re-enter information manually. This creates additional labor requirements and increases the possibility of human error.
Duplicated data can also create conflicting versions of information. If customer, employee, inventory or financial information exists in several systems without effective synchronization, different departments may operate using different versions of reality.
Interoperability is also closely connected to data governance. When information moves between systems, organizations must understand which system is authoritative, how data are transformed, who is responsible for correcting errors and how changes are communicated.
Cybersecurity must also be considered. Increasing interoperability can increase connectivity, but increased connectivity can also expand the attack surface. Every connection between systems should therefore be evaluated in terms of authentication, authorization, encryption, monitoring and incident response.
Privacy considerations become important when systems exchange personal or sensitive information. Interoperability should not result in unrestricted data sharing. Organizations should establish appropriate access controls, purpose limitations and data-sharing arrangements.
Interoperability also has strategic implications. A highly interoperable technology environment may allow an organization to adopt new applications more easily and replace individual components without redesigning the entire infrastructure. Poor interoperability can instead increase dependence on particular suppliers.
This creates a connection between interoperability and vendor lock-in. Proprietary systems that cannot easily communicate with alternative technologies may make future migration difficult. Interoperability should therefore be considered during technology selection rather than only after implementation.
Interoperability also affects organizational resilience. If one system fails, interoperable architectures may allow organizations to redirect processes to alternative systems. However, excessive dependence on a single integration mechanism may create a single point of failure.
A useful evaluation sequence is:
System requirements → data structures → communication mechanisms → semantic compatibility → security controls → organizational processes → interoperability testing → monitoring → change management.
The objective is not to maximize interoperability in every circumstance. Some systems may appropriately remain isolated because of security, safety or regulatory requirements. The evaluation question should therefore be whether the degree of interoperability is appropriate for the intended purpose and risk environment.
Interoperability should consequently be evaluated as a balance among connectivity, functionality, security, privacy, flexibility, cost and organizational control.
11.9 APIs and Integration
Application programming interfaces allow systems to communicate.
Evaluation should examine:
authentication;
reliability;
versioning;
documentation;
security;
data standards;
rate limitations;
error handling.
Poor integration design can create hidden organizational dependencies.
APIs have become fundamental components of modern digital ecosystems. They allow applications, databases, cloud services, IoT platforms, payment systems, artificial intelligence services and external providers to exchange information and functionality without requiring the underlying systems to be completely redesigned.
The strategic importance of APIs is therefore greater than their technical appearance might initially suggest. An API can become the connection through which an organization depends on an external service for a critical business process. If that interface changes, becomes unavailable or becomes prohibitively expensive, the organization may experience significant operational disruption.
Authentication determines how the system establishes the identity of an application, user or service requesting access. Evaluation should consider whether authentication mechanisms are sufficiently strong for the sensitivity of the information and operations involved.
Authorization determines what an authenticated entity is permitted to do. Effective API security should follow the principle of least privilege so that an application receives only the permissions required for its legitimate function.
Reliability should be evaluated under normal and abnormal operating conditions. An API may function correctly during ordinary usage but fail under high demand, network disruption or unusual input conditions. Evaluation should therefore include response times, availability, capacity and failure behavior.
Versioning is important because APIs evolve. A provider may modify data structures, authentication mechanisms, endpoints or functionality. If changes are not managed carefully, dependent applications may stop functioning.
Organizations should therefore understand how API versions are introduced, supported and retired. A technology that depends on an API that can be discontinued without sufficient notice may create significant operational risk.
Documentation is another important factor. Poor documentation increases the time and expertise required to integrate systems and makes future maintenance more difficult. Documentation should describe available functions, data formats, authentication requirements, error conditions and version histories.
Security should be evaluated throughout the API lifecycle. This includes authentication, authorization, encryption, input validation, logging, rate controls, vulnerability management and monitoring. APIs can become attractive targets because they often provide direct pathways into data and organizational functions.
Data standards are important because successful integration depends on common interpretations of information. An API that transfers data in a technically readable format may still create problems if field definitions, units, codes or classifications differ.
Rate limitations can protect systems from excessive requests and denial-of-service conditions, but they can also affect business operations if limits are too restrictive. Organizations should understand how rate limits are applied and what happens when limits are exceeded.
Error handling should also be evaluated. A resilient integration should provide meaningful error information without exposing sensitive technical details. Systems should be able to distinguish between temporary failures, authentication problems, invalid requests and permanent conditions.
Integration dependencies should be mapped across the organization. An apparently minor API may support numerous downstream systems. If that API fails, several business processes may be affected simultaneously.
This creates the possibility of cascading failure. A failure in an external API can prevent an internal application from obtaining data, which can prevent another system from operating, which can subsequently affect customers or employees.
Integration testing should therefore examine not only whether systems work individually but whether they continue to work when connected systems behave unexpectedly.
A useful sequence is:
System requirement → API selection → authentication → authorization → data exchange → validation → error handling → monitoring → version management → dependency review → exit or replacement planning.
API evaluation should also consider contractual and governance issues. Organizations should understand service-level agreements, data ownership, permitted uses, liability arrangements, change-notification procedures and termination conditions.
The central principle is that integration should create controlled interoperability rather than uncontrolled dependency. APIs should therefore be evaluated as strategic infrastructure rather than merely as technical interfaces.
11.10 Digital Platforms
Digital platforms connect different groups of users.
Examples include:
marketplaces;
social networks;
educational platforms;
financial platforms;
application ecosystems.
Platform value can increase through network effects, where the value to one user increases as participation by other users grows (Parker et al., 2016; Tiwana et al., 2010).
However, network effects can also create market concentration and dependency.
Digital platforms differ from conventional software applications because they often facilitate interactions among multiple groups rather than serving a single category of user. A marketplace may connect buyers and sellers. An educational platform may connect students, instructors, institutions and content providers. A financial platform may connect customers, merchants, financial institutions and service providers.
The value of the platform therefore depends partly on the quality of the relationships it facilitates.
Platform evaluation should begin with its purpose and stakeholder groups. The evaluator should identify who provides information, who consumes information, who performs transactions, who controls the platform and who bears the consequences of platform decisions.
Network effects can create substantial growth because participation by one group can increase the value available to other groups. More sellers may attract more buyers, while more buyers may attract more sellers. Similar dynamics can occur in social, financial, educational and application ecosystems.
However, network effects can also produce concentration. Once a platform becomes widely adopted, users may remain on it because other participants are already there. This can create switching barriers even when users are dissatisfied with particular aspects of the platform.
Platform governance therefore becomes an important evaluation dimension. Platform operators may establish rules concerning participation, content, transactions, data use, pricing, ranking, access and dispute resolution.
These rules can significantly affect stakeholders. A change in platform algorithms may alter which products users see, which educational resources students’ access or which service providers receive visibility.
Algorithmic governance is consequently an important component of platform evaluation. Where algorithms determine rankings, recommendations, access or visibility, evaluators should consider transparency, fairness, accountability and the possibility of unintended effects.
Data concentration is another important issue. Platforms may accumulate extensive information about users, transactions, preferences and interactions. This can create economic value while simultaneously creating privacy, cybersecurity and governance risks.
Platforms may also become dependent on complementary participants. Developers may build applications on a platform, merchants may rely on a marketplace, or educational institutions may depend on a learning platform. Changes in platform policies can therefore affect entire ecosystems rather than individual users.
Economic evaluation should examine both value creation and value distribution. A platform may create substantial overall economic activity while distributing benefits and costs unevenly among participants.
Interoperability is particularly important in platform environments. The ability to export data, communicate with alternative platforms and use external services can affect the strategic flexibility of participants.
Digital platforms should also be evaluated over time. Early-stage platform performance may differ substantially from mature-stage performance. As a platform grows, governance challenges, cybersecurity requirements, regulatory scrutiny and market concentration may increase.
The evaluation should therefore consider the platform lifecycle:
Platform purpose → participant recruitment → network development → data accumulation → governance → scaling → market dependency → regulatory and stakeholder pressures → adaptation.
A platform can therefore generate substantial technological and economic value while simultaneously creating strategic dependency. The evaluation should not treat either outcome as sufficient by itself.
The central question is whether the platform creates sustainable value for its relevant stakeholders while maintaining appropriate governance, security, interoperability, accountability and flexibility.
11.11 Network Effects
Network effects can produce rapid growth.
But evaluation should also consider:
market power;
switching costs;
data concentration;
interoperability;
platform governance;
dependence on platform rules.
A platform can therefore generate significant economic value while creating strategic dependency.
Network effects occur when the value of a product, service or platform changes as the number or characteristics of other users’ changes. Direct network effects occur when additional users directly increase value for existing users. Indirect network effects can arise when growth in one participant group increases value for another group.
The presence of network effects can significantly influence technology adoption. Users may adopt a platform not simply because of its intrinsic technical characteristics but because other users, suppliers, developers or organizations already use it.
This creates an important distinction between intrinsic technological value and ecosystem value. A platform may have technically similar functionality to another platform but may be more useful because it provides access to a larger ecosystem.
However, the same mechanism that supports growth can also create dependency. Once users have invested time, money, data and relationships in a particular platform, leaving it may become increasingly difficult.
Switching costs can be financial, technical, contractual, organizational or social. Financial costs may involve migration expenses. Technical costs may involve data conversion or system redevelopment. Organizational costs may involve employee retraining. Social costs may arise when users depend on relationships established through the platform.
Data concentration can reinforce network effects. A platform that accumulates large quantities of information may use that information to improve services, personalize interactions or develop new products. Competitors may have difficulty reproducing the same data resources.
This can create feedback loops:
More users → more data → improved service → greater attractiveness → more users.
Although such feedback can support innovation and service improvement, it may also create barriers to entry for competitors.
Market power should therefore be considered as part of technology evaluation where network effects are significant. The evaluation need not assume that concentration is automatically harmful, but it should identify how concentration affects users, suppliers, competitors and organizational resilience.
Interoperability can reduce some forms of dependency by allowing users and organizations to connect with alternative services. Data portability can similarly reduce switching costs when users can transfer information in usable formats.
Platform governance is therefore central. Participants may depend on rules established by platform operators concerning pricing, access, content, ranking, advertising, data use, account suspension and dispute resolution.
Changes to these rules can have substantial consequences. Organizations that depend heavily on a platform should therefore monitor not only technical performance but also changes in platform governance.
Network effects can also produce resilience and vulnerability simultaneously. A large ecosystem may provide redundancy and extensive resources, but it may also create systemic dependence on a common infrastructure.
For example, if a large number of organizations depend on the same platform for communication, payment processing, cloud services or identity management, an outage may affect many organizations simultaneously.
Network-effect evaluation should therefore consider both individual and ecosystem-level consequences.
The appropriate evaluation sequence is:
Network structure → participant dependence → value creation → switching costs → data concentration → interoperability → governance → market concentration → resilience → strategic alternatives.
The objective is not to eliminate network effects. Network effects are often central to the usefulness of digital platforms. The objective is to understand how network effects affect technological value, organizational autonomy, competition, resilience and stakeholder outcomes.
Within the broader multidimensional technology evaluation framework, network effects should therefore be treated as an important relationship between technical capability, economic value, governance and systemic risk.
11.12 Vendor Lock-In
Vendor lock-in occurs when switching away from a technology provider becomes difficult or expensive.
Evaluation should examine:
data portability;
contractual restrictions;
proprietary formats;
migration costs;
technical compatibility;
alternative providers.
Exit planning should be part of technology selection rather than an afterthought.
Vendor lock-in is an important strategic risk because technology decisions frequently create long-term dependencies. An organization may initially select a provider because of price, functionality, performance or convenience and only later discover that replacing the provider is difficult.
Lock-in can occur through technical, contractual, economic, organizational and data-related mechanisms. These mechanisms may operate independently or reinforce one another.
Data portability is one of the most important considerations. Organizations should determine whether their information can be exported in complete, accurate and usable formats. A provider may technically permit data export while making the process sufficiently complex that migration becomes impractical.
Proprietary formats can create additional dependency. If data are stored in formats that alternative systems cannot easily interpret, migration may require expensive conversion or redevelopment.
Contractual restrictions may also create lock-in. Long-term commitments, termination fees, minimum purchase requirements or restrictions on data movement can limit organizational flexibility.
Migration costs should be considered before adoption. These may include software redevelopment, data conversion, infrastructure changes, employee training, consulting, testing, temporary parallel operation and business disruption.
Technical compatibility is also important. An organization may have developed numerous integrations around a particular provider. Even if the central service can be replaced, dependent applications may require extensive modification.
Alternative providers should therefore be identified during technology selection. The existence of realistic alternatives can strengthen bargaining power and reduce dependency.
Vendor lock-in is not necessarily undesirable in every circumstance. Long-term relationships can produce integration benefits, specialized expertise, reliability and economies of scale. The evaluation question is therefore whether the level of dependency is proportionate to the benefits obtained.
An organization should distinguish between deliberate strategic dependency and accidental dependency. Deliberate dependency occurs when an organization knowingly accepts reliance on a provider because the expected benefits justify it. Accidental dependency occurs when switching difficulties emerge without having been considered during the original technology decision.
Exit planning should therefore be incorporated into the initial business and technology case.
An effective exit strategy should consider what would happen if the provider increased prices, discontinued a service, changed contractual conditions, suffered a major outage, experienced a security incident, ceased operations or became incompatible with organizational requirements.
The analysis should also consider the time required to exit. A system that can technically be replaced in theory may still require months or years to migrate in practice. Critical organizations should therefore assess whether the transition period is compatible with their operational risk tolerance.
Multi-vendor strategies may reduce dependency in some circumstances, but they can also increase complexity, integration requirements and management costs. Diversification should therefore be evaluated rather than automatically assumed to be beneficial.
Open standards can support portability and interoperability, although their presence does not guarantee easy migration. Organizations must determine whether standards are actually implemented in a way that allows practical transfer.
Contractual arrangements should also establish appropriate rights concerning data access, data return, service termination, transition assistance, security incidents and business continuity.
Vendor risk should be evaluated throughout the lifecycle. A provider that is financially stable and strategically appropriate at the beginning of a technology project may later change ownership, business strategy, pricing structure or service architecture.
Continuous monitoring is therefore required.
A useful sequence is:
Technology requirement → vendor selection → dependency assessment → interoperability assessment → contractual review → data portability assessment → migration planning → operational monitoring → alternative-provider review → exit readiness.
Exit planning should not imply that an organization expects the technology relationship to fail. Rather, it demonstrates that the organization has considered continuity and strategic autonomy.
The deeper principle is that technological adoption should not eliminate organizational choice unnecessarily. A technology decision should create value while preserving an appropriate level of flexibility to respond to future technological, economic, regulatory and organizational changes.
Vendor lock-in therefore connects directly with interoperability, APIs, cloud computing, digital platforms, data governance and systemic risk. A technology that performs well today may still create significant long-term risk if the organization cannot reasonably change, replace, modify or discontinue it when circumstances require.
Within the multidimensional technology evaluation framework, exit capability can therefore be treated as an indicator of technological resilience and strategic control. The final evaluation question is not merely whether an organization can successfully adopt a technology, but whether it can continue, modify, migrate or exit that technology in a controlled manner throughout its lifecycle.