Data Engineering

Data Engineering has become increasingly relevant in the highly-connected, AI driven world. Today, data architectures are as vast and varied as the use-cases they support. For most of us, data architecture simply meant running an RDBMS for all of our needs, from transactional read-write workloads to ad-hoc point and scan analytics loads. As data grew in numbers more use-cases for data-driven products such as fraud detection systems, recommender systems, personalization services came into existence.

Data engineers entered the field to solve our problems by introducing specialized data stores for example; search engines, graph engines, large scale data processing such as Spark, NoSQL, stream processing and the machinery to glue them together like ETL pipelines, Kafka, Sqoop, Flume.