Data-intensive computing involves a cluster of distributed, potentially virtualized servers. Much like an operating system manages resources on your computer, a system needs to oversee the requirements and server resources in the cluster. It identifies, among other functions, which servers have available resources to install and run new programs.

In data engineering, it’s not just about the program itself. The program, along with all its dependencies, are packaged together as a container and installed. This program, also known as an application or service, collaborates with other services, which can be instances of the same service or different services.

In such a distributed-computing environment, where every application  is interconnected through a computer network, it’s crucial to understand concepts like reliability, scalability, maintainability, and security. These are inherent design considerations.

Addressing the ever-growing quantity, complexity, and importance of data requires new strategies—algorithms, design principles, and tools—that enable you to handle data delivery “at least once,” “exactly once,” or “at most once” in an economically justifiable manner. 

Cost monitoring is crucial because, without proper care, expenses can increase rapidly. Applications  and servers must also reveal metrics, enabling the monitoring system to signal whether the system is operating as anticipated.

Finally, as data engineering involves processing vast amounts of potentially privacy – and security – sensitive data, addressing security is essential.