Principal Data Engineer - Cloud Data Pipelines & DataOps

Dublin 1, Ireland Posted on 06/17/2024

As a Principal DataEngineer, your responsibilities will include:

• Design and build data pipelinesto process terabytes of data
  
• Orchestrate in Airflow the datatasks to run on Kubernetes/Hadoop for the ingestion, processing andcleaning of data.
  
• Create Docker images forvarious applications and deploy them on Kubernetes
  
• Design and build best in classprocesses to clean and standardize data.
  
• Troubleshoot production issues inour Elastic Environment
  
• Tuning and optimizing dataprocesses
  

Advancing the team's DataOps culture (CI/CD,Orchestration, Testing, Monitoring) and building out standard developmentpatterns

• Drive innovation bytesting new technology and approaches to continually advance thecapability of the data engineering function.
  
• Drive efficiencies incurrent engineering processes via standardization and migration ofexisting on-premise processes to the cloud
  
• Ensuring DataQuality - building best in class data quality monitoring thatensure that all data products exceed customer expectations.
  

Required Qualifications:

• Computer Science bachelor'sdegree or similar.
  
• Good understanding of Data Modelling techniques i.e.DataVault, Kimble Star
  
• Excellent understanding of Column-Store RDBMS(DataBricks, Snowflake, Redshift, Vertica, Clickhouse)
  
• Good experience handling real-time, near real-timeand batch data ingestions
  
• Hands on experience on thefollowing technologies:
  
• Developing processes in Spark
  
• Exposure to Kubernetes andLinux containers (i.e. Docker)
  
• Related/complementary opensource software platforms and languages (e.g. Scala, Python, Java, Linux)
  
• Proven trackrecord of designing effective data strategies and leveraging modern dataarchitectures that resulted in business value
  
• Experiencebuilding cloud-native data pipelines on either AWS, Azure or GCP,following best practices in cloud deployments
  

Strong DataOpsexperience (CI/CD,Orchestration, Testing, Monitoring)

• Strongexperience leading and developing data engineering teams
  

Demonstrated effective interpersonal,influence, collaboration and listening skills

Excellent time management, organizational andprioritization skills with ability to balance multiple priorities.

Preferred Qualifications:

Experience with data tokenization and different techniques andtools i.e. DataVant, Protegrity

Experience with Azure Data Factory, Databricks and Snowflake

• Experiencewith Apache Spark and related Big Data stack and technologies, PySparkScala
  

Experience working with Apache Kafka, building appropriateproducer/consumer apps

Experience working with Kubernetes and Docker, and knowledgeableabout cloud infrastructure automation and management (e.g., Terraform)

Experience working in projects with agile/scrum methodologies

Familiarity with production quality ML and/or AI model developmentand deployment.

Healthcare industry knowledge and experience with exposure to EDI,HIPAA, HL7 and FHIR integration standards