Data Engineer - Global Insurance firm

Randstad is partnered with global Life Insurance client in their search for a Data Engineer. Through this role you will join the firm's global Data Management department, where you will work alongside brilliant data professionals. If you have strong background in Data Engineering with Scala/Python/Java, Spark , and Databricks this role is for you. 

Responsibilities

• Analyze business requirements to create scalable data solutions that meet functional and non-functional requirements.

• Lead the end-to-end migration of legacy Synapse T-SQL workloads and pipelines into optimized Spark/PySpark jobs and Databricks Workflows.

• Define Lakehouse architecture, storage, and security models integrating Azure Databricks, ADLS, Azure Cosmos DB, and REST APIs.

• Design and implement highly scalable batch and streaming data pipelines in Scala/Spark/Python using a Medallion Architecture.

• Create and maintain reusable frameworks for data ingestion, validation, and transformation leveraging Delta Lake capabilities (schema evolution, time travel, Z-ordering).

• Implement and manage Databricks Unity Catalog for centralized data governance, security modeling, and maintaining data lineage.

• Drive new Databricks-native initiatives, including Delta Live Tables (DLT) for declarative pipeline management and Databricks SQL for high-performance BI serving.

• Automate unit testing and release processes with CI/CD pipelines, utilizing tools like Databricks Repos or Databricks Asset Bundles.

• Create scripts for operations, releases, data migrations, and validations, orchestrating complex dependencies.

• Provide technical support for releases and incidents, focusing on Spark cluster tuning, compute right-sizing, and cost optimization.

Required Skills & Qualifications

• Minimum 5+ years of application development or data engineering experience.

• Strong programming experience in Python, Java, or Spark, with a strong willingness to learn and develop in Scala.

• Experience in relational databases and advanced SQL, essential for understanding and migrating legacy Synapse/T-SQL workloads.

• Hands-on experience with Azure Databricks, including migrating or building solutions using Delta Lake, Unity Catalog, and Databricks Workflows.

• Familiarity with data engineering DevOps practices, CI/CD pipeline automation, and deployment tools (e.g., Databricks Repos, DABs).

Good-to-Have (Preferred Qualifications)

• Proven experience building and optimizing data pipelines to extract, transform, and load (ETL/ELT) large-scale data.

• Broad knowledge of Microsoft Azure ecosystem or other public cloud architectures (storage, security, networking).

• Hands-on experience using orchestration and ETL tools like Azure Data Factory (ADF).

• Working knowledge of REST APIs and API-based data integration architectures.