Big Data Engineering, Cloud Analytics and Scalable AI
Free to read online. For a copy of this book, or to quote, highlight or reproduce any part of it, please request permission through the contact page beforehand.
Editorial and publication note
This is a proposed educational series. IEEE-style technical writing and citations do not imply IEEE endorsement, publication, accreditation, certification, or affiliation. Any standard, designation, author credential, publication detail, or institutional relationship must be verified before publication. These Word files are expanded educational drafts and chapter-development manuscripts for editorial review; they are not represented as final peer-reviewed or publication-ready books.
The proposed series title is descriptive only. Do not use IEEE logos or imply IEEE affiliation without authorisation. Any references to standards should be checked in the official IEEE Standards Association catalogue and used only within their stated scope.
About this book
Covers data architecture, storage, pipelines, batch and streaming, distributed computing, cloud platforms, MLOps, monitoring, security, governance, and cost.
Who should use this book
This volume is designed for data engineers, software developers, and architects. Instructors should adapt pacing, depth, tools, and examples to learner needs and local curriculum requirements.
Learning outcomes
- Explain the main concepts and vocabulary introduced throughout the volume.
- Apply a structured method to a relevant practical problem.
- Interpret evidence, identify limitations, and communicate results clearly.
- Consider privacy, fairness, safety, security, and human oversight where relevant.
How to use this book
Read the chapter explanation, complete the activity, answer the review questions, and keep a learning journal recording assumptions, decisions, mistakes, and improvements. For technical work, save code, data dictionaries, test results, and software-version details.
Table of contents
- Chapter 1: The Characteristics of Big Data
- Chapter 2: Relational and Non-Relational Databases
- Chapter 3: Data Warehouses, Lakes, and Lakehouses
- Chapter 4: ETL and ELT Data Pipelines
- Chapter 5: Batch and Stream Processing
- Chapter 6: Distributed Computing Principles
- Chapter 7: Cloud Infrastructure and Storage
- Chapter 8: Data Integration and Interoperability
- Chapter 9: Data Quality and Metadata Management
- Chapter 10: Feature Pipelines and Model Data
- Chapter 11: Model Deployment and MLOps
- Chapter 12: Monitoring, Logging, and Data Drift
- Chapter 13: Security and Access Management
- Chapter 14: Reliability, Cost, and Scalability
- Chapter 15: Data Governance and Lifecycle Management
- Chapter 16: Scalable Data Platform Project
Chapter 1: The Characteristics of Big Data
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in the characteristics of big data.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Large-scale data work may involve volume, velocity, variety, reliability, and value trade-offs. Distributed architecture is justified when workload, availability, or throughput needs require it; complexity itself is not a goal.
Worked example and application
Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.
Practical activity
Estimate a fictional workload and decide whether a simple database is sufficient.
Chapter review
- What is the main idea in 'The Characteristics of Big Data'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 2: Relational and Non-Relational Databases
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in relational and non-relational databases.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Relational systems use tables, schemas, keys, and query languages; non-relational systems support other access patterns and data models. Choose based on consistency, query, scale, and operational needs.
Worked example and application
Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.
Practical activity
Model a small customer-order dataset and identify primary and foreign keys.
Chapter review
- What is the main idea in 'Relational and Non-Relational Databases'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 3: Data Warehouses, Lakes, and Lakehouses
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in data warehouses, lakes, and lakehouses.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Warehouses commonly support governed analytics; data lakes store varied raw data; lakehouse designs combine aspects of both. Architecture choices affect discoverability, quality, access control, and cost.
Worked example and application
Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.
Practical activity
Compare three architecture options for a fictional analytics team.
Chapter review
- What is the main idea in 'Data Warehouses, Lakes, and Lakehouses'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 4: ETL and ELT Data Pipelines
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in etl and elt data pipelines.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
ETL transforms data before loading; ELT loads data before transformation in the target platform. Pipelines need schema checks, retries, idempotency, lineage, and alerting.
Worked example and application
Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.
Practical activity
Design a pipeline with source, validation, transformation, destination, and failure handling.
Chapter review
- What is the main idea in 'ETL and ELT Data Pipelines'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 5: Batch and Stream Processing
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in batch and stream processing.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Batch processes bounded groups of records; streaming processes events as they arrive. Latency, event ordering, late arrivals, duplicate events, and delivery guarantees influence design.
Worked example and application
Worked example: define a real-world task, list the inputs and constraints, sketch the steps or system components, and test the proposed solution with at least one ordinary case and one edge case. Record the outcome and what you would improve.
Practical activity
Choose a processing pattern for monthly reporting and for real-time equipment alerts.
Chapter review
- What is the main idea in 'Batch and Stream Processing'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 6: Distributed Computing Principles
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in distributed computing principles.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Distributed systems partition work across nodes and must account for communication, partial failure, consistency, and coordination. Parallelism can improve throughput but also creates operational complexity.
Worked example and application
Worked example: define a real-world task, list the inputs and constraints, sketch the steps or system components, and test the proposed solution with at least one ordinary case and one edge case. Record the outcome and what you would improve.
Practical activity
Draw a partitioned data-processing job and identify failure and recovery points.
Chapter review
- What is the main idea in 'Distributed Computing Principles'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 7: Cloud Infrastructure and Storage
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in cloud infrastructure and storage.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Cloud systems provide on-demand compute, storage, managed services, and network capabilities. Design must consider identity, region, resilience, data transfer, lock-in, and cost controls.
Worked example and application
Worked example: define a real-world task, list the inputs and constraints, sketch the steps or system components, and test the proposed solution with at least one ordinary case and one edge case. Record the outcome and what you would improve.
Practical activity
Prepare a cloud architecture sketch and a basic cost/risk checklist.
Chapter review
- What is the main idea in 'Cloud Infrastructure and Storage'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 8: Data Integration and Interoperability
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in data integration and interoperability.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Integration combines data from systems with different formats, meanings, identifiers, and update schedules. Shared definitions and contracts reduce silent incompatibility.
Worked example and application
Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.
Practical activity
Define a data contract for a fictional sales event, including schema and change policy.
Chapter review
- What is the main idea in 'Data Integration and Interoperability'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 9: Data Quality and Metadata Management
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in data quality and metadata management.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Metadata describes datasets, meaning, ownership, lineage, freshness, and permitted use. Quality checks should reflect business meaning, not only technical validity.
Worked example and application
Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.
Practical activity
Specify quality tests and metadata fields for a dataset used in management reporting.
Chapter review
- What is the main idea in 'Data Quality and Metadata Management'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 10: Feature Pipelines and Model Data
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in feature pipelines and model data.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Feature pipelines transform raw inputs into model-ready variables. Training-serving skew, leakage, freshness, and inconsistent definitions can undermine deployed models.
Worked example and application
Worked example: imagine a small team considering this concept for a clearly scoped, low-risk task. First define the intended outcome and inputs; then choose a simple baseline, document assumptions, test representative examples, inspect errors, and explain when the method should not be used. Compare the result with a human-reviewed alternative before proposing deployment.
Practical activity
Draw a feature pipeline and explain how training and production features remain consistent.
Chapter review
- What is the main idea in 'Feature Pipelines and Model Data'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 11: Model Deployment and MLOps
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in model deployment and mlops.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
MLOps connects model development, versioning, testing, deployment, monitoring, and retraining. Automating deployment does not remove the need for approval gates and rollback.
Worked example and application
Worked example: imagine a small team considering this concept for a clearly scoped, low-risk task. First define the intended outcome and inputs; then choose a simple baseline, document assumptions, test representative examples, inspect errors, and explain when the method should not be used. Compare the result with a human-reviewed alternative before proposing deployment.
Practical activity
Design a release workflow with automated tests, human approval, canary deployment, and rollback.
Chapter review
- What is the main idea in 'Model Deployment and MLOps'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 12: Monitoring, Logging, and Data Drift
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in monitoring, logging, and data drift.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Monitor system health, data quality, prediction behaviour, and outcome metrics. Drift signals change but does not automatically prove that a model has become unusable.
Worked example and application
Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.
Practical activity
Draft monitoring thresholds and an escalation plan for a changing data distribution.
Chapter review
- What is the main idea in 'Monitoring, Logging, and Data Drift'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 13: Security and Access Management
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in security and access management.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Least privilege, identity controls, encryption, secrets management, network boundaries, and audit logs protect data systems. Access should be granted according to role and legitimate need.
Worked example and application
Worked example: list the system's users, affected people, assets, likely failure modes, and potential consequences. Select preventive and detective controls, assign an accountable owner, and record the residual risk that remains after controls.
Practical activity
Create a role-based access matrix for analysts, engineers, and administrators.
Chapter review
- What is the main idea in 'Security and Access Management'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 14: Reliability, Cost, and Scalability
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in reliability, cost, and scalability.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Reliability engineering balances availability, recovery objectives, redundancy, and cost. Capacity plans should be tested against realistic load and failure conditions.
Worked example and application
Worked example: define a real-world task, list the inputs and constraints, sketch the steps or system components, and test the proposed solution with at least one ordinary case and one edge case. Record the outcome and what you would improve.
Practical activity
Compare two architecture options using cost, reliability, latency, and maintenance criteria.
Chapter review
- What is the main idea in 'Reliability, Cost, and Scalability'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 15: Data Governance and Lifecycle Management
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in data governance and lifecycle management.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
Governance assigns responsibility for quality, access, retention, deletion, and acceptable use. Lifecycle controls help ensure data is not retained indefinitely without purpose.
Worked example and application
Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.
Practical activity
Create a data lifecycle policy covering collection, use, retention, access, and deletion.
Chapter review
- What is the main idea in 'Data Governance and Lifecycle Management'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Chapter 16: Scalable Data Platform Project
Learning outcomes
By the end of this chapter, the learner should be able to:
- Explain the central concepts introduced in scalable data platform project.
- Apply the ideas to a clearly defined example or practical task.
- Identify at least one limitation, assumption, or responsible-use consideration.
Core explanation
An end-to-end platform design links ingestion, storage, transformation, analytics, model operations, governance, and monitoring.
Worked example and application
Worked example: use a small, fictional dataset with a documented meaning for each column. Check completeness and units, calculate a simple summary, visualise the result, and write one conclusion that is directly supported by the observations. State what the dataset cannot establish.
Practical activity
Design an end-to-end data platform, including architecture, data quality controls, access policies, monitoring, and cost considerations.
Chapter review
- What is the main idea in 'Scalable Data Platform Project'?
- Which assumptions or data requirements matter most in this topic?
- What could cause the approach to fail or produce a misleading result?
- How would you explain the result or limitation to a non-specialist?
Key takeaways
- Start with a clear question, objective, or system requirement.
- Document inputs, assumptions, methods, and limitations.
- Test the approach with relevant examples and review the consequences of errors.
Capstone project
Design an end-to-end data platform, including architecture, data quality controls, access policies, monitoring, recovery, and cost considerations. Include an architecture diagram, data contracts, operational runbook, and governance summary.
Suggested project deliverables
- A short problem statement and intended audience.
- A description of data, inputs, assumptions, tools, and method.
- Evidence of testing, analysis, or evaluation appropriate to the project.
- A clear explanation of results, limitations, and foreseeable risks.
- A final report or presentation and a reflection on possible improvements.
Assessment guidance
Assess architectural fit, data quality, failure handling, security, monitoring, lifecycle governance, and realistic cost/reliability trade-offs.
Glossary
ETL: Extract, transform, load: a data integration pattern.
Data lineage: A record of where data originated and how it moved or changed.
MLOps: Practices for operationalising machine-learning systems across their lifecycle.
Data drift: A change in input data distributions or relationships over time.
Idempotency: A property whereby repeating an operation has the same effect as performing it once, within defined conditions.
Responsible practice checklist
- Use data and materials only when their source and permitted use are understood.
- Avoid exposing personal, confidential, or credential information.
- Document assumptions, missing information, and important limitations.
- Test ordinary cases, boundary cases, and foreseeable failure conditions.
- Use human review where errors could materially affect people, safety, rights, or resources.
- Do not claim that a system is fair, safe, secure, compliant, or reliable without appropriate evidence.
References and further reading
The source material supplied for this project does not provide a complete, volume-specific bibliography. Before publication, add verified primary sources, textbooks, official documentation, and standards applicable to this volume. Confirm exact titles, editions, publication dates, URLs, licences, and access dates. Do not cite a standard as evidence of conformity unless the required assessment has actually been performed.
For IEEE-style technical manuscripts, use numbered citations in order of first appearance when required by the target venue, and verify the venue's official author instructions.