Prof. Dr. Larry AdamsAcademic, Author & Researcher
The IEEE Series: AI, Machine Learning & Data Science · Book 8

Big Data Engineering, Cloud Analytics and Scalable AI

Intended audience: Data engineers, software developers, and architects

Dr. Lance Adams and Prof. Dr. Larry Adams

Free to read online. For a copy of this book, or to quote, highlight or reproduce any part of it, please request permission through the contact page beforehand.

Editorial and publication note

This is a proposed educational series. IEEE-style technical writing and citations do not imply IEEE endorsement, publication, accreditation, certification, or affiliation. Any standard, designation, author credential, publication detail, or institutional relationship must be verified before publication. These Word files are expanded educational drafts and chapter-development manuscripts for editorial review; they are not represented as final peer-reviewed or publication-ready books.

The proposed series title is descriptive only. Do not use IEEE logos or imply IEEE affiliation without authorisation. Any references to standards should be checked in the official IEEE Standards Association catalogue and used only within their stated scope.

About this book

Covers data architecture, storage, pipelines, batch and streaming, distributed computing, cloud platforms, MLOps, monitoring, security, governance, and cost.

Who should use this book

This volume is designed for data engineers, software developers, and architects. Instructors should adapt pacing, depth, tools, and examples to learner needs and local curriculum requirements.

Learning outcomes

  • Explain the main concepts and vocabulary introduced throughout the volume.
  • Apply a structured method to a relevant practical problem.
  • Interpret evidence, identify limitations, and communicate results clearly.
  • Consider privacy, fairness, safety, security, and human oversight where relevant.

How to use this book

Read the chapter explanation, complete the activity, answer the review questions, and keep a learning journal recording assumptions, decisions, mistakes, and improvements. For technical work, save code, data dictionaries, test results, and software-version details.

Table of contents

  • Chapter 1: The Characteristics of Big Data
  • Chapter 2: Relational and Non-Relational Databases
  • Chapter 3: Data Warehouses, Lakes, and Lakehouses
  • Chapter 4: ETL and ELT Data Pipelines
  • Chapter 5: Batch and Stream Processing
  • Chapter 6: Distributed Computing Principles
  • Chapter 7: Cloud Infrastructure and Storage
  • Chapter 8: Data Integration and Interoperability
  • Chapter 9: Data Quality and Metadata Management
  • Chapter 10: Feature Pipelines and Model Data
  • Chapter 11: Model Deployment and MLOps
  • Chapter 12: Monitoring, Logging, and Data Drift
  • Chapter 13: Security and Access Management
  • Chapter 14: Reliability, Cost, and Scalability
  • Chapter 15: Data Governance and Lifecycle Management
  • Chapter 16: Scalable Data Platform Project

Chapter 1: The Characteristics of Big Data

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in the characteristics of big data.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Large-scale data work may involve volume, velocity, variety, reliability, and value trade-offs. Distributed architecture is justified when workload, availability, or throughput needs require it; complexity itself is not a goal.

Worked example and application

Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.

Practical activity

Estimate a fictional workload and decide whether a simple database is sufficient.

Chapter review

  • What is the main idea in 'The Characteristics of Big Data'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 2: Relational and Non-Relational Databases

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in relational and non-relational databases.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Relational systems use tables, schemas, keys, and query languages; non-relational systems support other access patterns and data models. Choose based on consistency, query, scale, and operational needs.

Worked example and application

Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.

Practical activity

Model a small customer-order dataset and identify primary and foreign keys.

Chapter review

  • What is the main idea in 'Relational and Non-Relational Databases'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 3: Data Warehouses, Lakes, and Lakehouses

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in data warehouses, lakes, and lakehouses.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Warehouses commonly support governed analytics; data lakes store varied raw data; lakehouse designs combine aspects of both. Architecture choices affect discoverability, quality, access control, and cost.

Worked example and application

Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.

Practical activity

Compare three architecture options for a fictional analytics team.

Chapter review

  • What is the main idea in 'Data Warehouses, Lakes, and Lakehouses'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 4: ETL and ELT Data Pipelines

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in etl and elt data pipelines.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

ETL transforms data before loading; ELT loads data before transformation in the target platform. Pipelines need schema checks, retries, idempotency, lineage, and alerting.

Worked example and application

Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.

Practical activity

Design a pipeline with source, validation, transformation, destination, and failure handling.

Chapter review

  • What is the main idea in 'ETL and ELT Data Pipelines'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 5: Batch and Stream Processing

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in batch and stream processing.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Batch processes bounded groups of records; streaming processes events as they arrive. Latency, event ordering, late arrivals, duplicate events, and delivery guarantees influence design.

Worked example and application

Worked example: define a real-world task, list the inputs and constraints, sketch the steps or system components, and test the proposed solution with at least one ordinary case and one edge case. Record the outcome and what you would improve.

Practical activity

Choose a processing pattern for monthly reporting and for real-time equipment alerts.

Chapter review

  • What is the main idea in 'Batch and Stream Processing'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 6: Distributed Computing Principles

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in distributed computing principles.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Distributed systems partition work across nodes and must account for communication, partial failure, consistency, and coordination. Parallelism can improve throughput but also creates operational complexity.

Worked example and application

Worked example: define a real-world task, list the inputs and constraints, sketch the steps or system components, and test the proposed solution with at least one ordinary case and one edge case. Record the outcome and what you would improve.

Practical activity

Draw a partitioned data-processing job and identify failure and recovery points.

Chapter review

  • What is the main idea in 'Distributed Computing Principles'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 7: Cloud Infrastructure and Storage

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in cloud infrastructure and storage.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Cloud systems provide on-demand compute, storage, managed services, and network capabilities. Design must consider identity, region, resilience, data transfer, lock-in, and cost controls.

Worked example and application

Worked example: define a real-world task, list the inputs and constraints, sketch the steps or system components, and test the proposed solution with at least one ordinary case and one edge case. Record the outcome and what you would improve.

Practical activity

Prepare a cloud architecture sketch and a basic cost/risk checklist.

Chapter review

  • What is the main idea in 'Cloud Infrastructure and Storage'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 8: Data Integration and Interoperability

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in data integration and interoperability.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Integration combines data from systems with different formats, meanings, identifiers, and update schedules. Shared definitions and contracts reduce silent incompatibility.

Worked example and application

Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.

Practical activity

Define a data contract for a fictional sales event, including schema and change policy.

Chapter review

  • What is the main idea in 'Data Integration and Interoperability'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 9: Data Quality and Metadata Management

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in data quality and metadata management.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Metadata describes datasets, meaning, ownership, lineage, freshness, and permitted use. Quality checks should reflect business meaning, not only technical validity.

Worked example and application

Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.

Practical activity

Specify quality tests and metadata fields for a dataset used in management reporting.

Chapter review

  • What is the main idea in 'Data Quality and Metadata Management'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 10: Feature Pipelines and Model Data

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in feature pipelines and model data.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Feature pipelines transform raw inputs into model-ready variables. Training-serving skew, leakage, freshness, and inconsistent definitions can undermine deployed models.

Worked example and application

Worked example: imagine a small team considering this concept for a clearly scoped, low-risk task. First define the intended outcome and inputs; then choose a simple baseline, document assumptions, test representative examples, inspect errors, and explain when the method should not be used. Compare the result with a human-reviewed alternative before proposing deployment.

Practical activity

Draw a feature pipeline and explain how training and production features remain consistent.

Chapter review

  • What is the main idea in 'Feature Pipelines and Model Data'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 11: Model Deployment and MLOps

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in model deployment and mlops.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

MLOps connects model development, versioning, testing, deployment, monitoring, and retraining. Automating deployment does not remove the need for approval gates and rollback.

Worked example and application

Worked example: imagine a small team considering this concept for a clearly scoped, low-risk task. First define the intended outcome and inputs; then choose a simple baseline, document assumptions, test representative examples, inspect errors, and explain when the method should not be used. Compare the result with a human-reviewed alternative before proposing deployment.

Practical activity

Design a release workflow with automated tests, human approval, canary deployment, and rollback.

Chapter review

  • What is the main idea in 'Model Deployment and MLOps'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 12: Monitoring, Logging, and Data Drift

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in monitoring, logging, and data drift.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Monitor system health, data quality, prediction behaviour, and outcome metrics. Drift signals change but does not automatically prove that a model has become unusable.

Worked example and application

Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.

Practical activity

Draft monitoring thresholds and an escalation plan for a changing data distribution.

Chapter review

  • What is the main idea in 'Monitoring, Logging, and Data Drift'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 13: Security and Access Management

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in security and access management.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Least privilege, identity controls, encryption, secrets management, network boundaries, and audit logs protect data systems. Access should be granted according to role and legitimate need.

Worked example and application

Worked example: list the system's users, affected people, assets, likely failure modes, and potential consequences. Select preventive and detective controls, assign an accountable owner, and record the residual risk that remains after controls.

Practical activity

Create a role-based access matrix for analysts, engineers, and administrators.

Chapter review

  • What is the main idea in 'Security and Access Management'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 14: Reliability, Cost, and Scalability

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in reliability, cost, and scalability.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Reliability engineering balances availability, recovery objectives, redundancy, and cost. Capacity plans should be tested against realistic load and failure conditions.

Worked example and application

Worked example: define a real-world task, list the inputs and constraints, sketch the steps or system components, and test the proposed solution with at least one ordinary case and one edge case. Record the outcome and what you would improve.

Practical activity

Compare two architecture options using cost, reliability, latency, and maintenance criteria.

Chapter review

  • What is the main idea in 'Reliability, Cost, and Scalability'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 15: Data Governance and Lifecycle Management

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in data governance and lifecycle management.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

Governance assigns responsibility for quality, access, retention, deletion, and acceptable use. Lifecycle controls help ensure data is not retained indefinitely without purpose.

Worked example and application

Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.

Practical activity

Create a data lifecycle policy covering collection, use, retention, access, and deletion.

Chapter review

  • What is the main idea in 'Data Governance and Lifecycle Management'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Chapter 16: Scalable Data Platform Project

Learning outcomes

By the end of this chapter, the learner should be able to:

  • Explain the central concepts introduced in scalable data platform project.
  • Apply the ideas to a clearly defined example or practical task.
  • Identify at least one limitation, assumption, or responsible-use consideration.

Core explanation

An end-to-end platform design links ingestion, storage, transformation, analytics, model operations, governance, and monitoring.

Worked example and application

Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.

Practical activity

Design an end-to-end data platform, including architecture, data quality controls, access policies, monitoring, and cost considerations.

Chapter review

  • What is the main idea in 'Scalable Data Platform Project'?
  • Which assumptions or data requirements matter most in this topic?
  • What could cause the approach to fail or produce a misleading result?
  • How would you explain the result or limitation to a non-specialist?

Key takeaways

  • Start with a clear question, objective, or system requirement.
  • Document inputs, assumptions, methods, and limitations.
  • Test the approach with relevant examples and review the consequences of errors.

Capstone project

Design an end-to-end data platform, including architecture, data quality controls, access policies, monitoring, recovery, and cost considerations. Include an architecture diagram, data contracts, operational runbook, and governance summary.

Suggested project deliverables

  • A short problem statement and intended audience.
  • A description of data, inputs, assumptions, tools, and method.
  • Evidence of testing, analysis, or evaluation appropriate to the project.
  • A clear explanation of results, limitations, and foreseeable risks.
  • A final report or presentation and a reflection on possible improvements.

Assessment guidance

Assess architectural fit, data quality, failure handling, security, monitoring, lifecycle governance, and realistic cost/reliability trade-offs.

Glossary

ETL: Extract, transform, load: a data integration pattern.

Data lineage: A record of where data originated and how it moved or changed.

MLOps: Practices for operationalising machine-learning systems across their lifecycle.

Data drift: A change in input data distributions or relationships over time.

Idempotency: A property whereby repeating an operation has the same effect as performing it once, within defined conditions.

Responsible practice checklist

  • Use data and materials only when their source and permitted use are understood.
  • Avoid exposing personal, confidential, or credential information.
  • Document assumptions, missing information, and important limitations.
  • Test ordinary cases, boundary cases, and foreseeable failure conditions.
  • Use human review where errors could materially affect people, safety, rights, or resources.
  • Do not claim that a system is fair, safe, secure, compliant, or reliable without appropriate evidence.

References and further reading

The source material supplied for this project does not provide a complete, volume-specific bibliography. Before publication, add verified primary sources, textbooks, official documentation, and standards applicable to this volume. Confirm exact titles, editions, publication dates, URLs, licences, and access dates. Do not cite a standard as evidence of conformity unless the required assessment has actually been performed.

For IEEE-style technical manuscripts, use numbered citations in order of first appearance when required by the target venue, and verify the venue's official author instructions.