Senior Apache Spark Technical Lead - Scala, Python

Senior Apache Spark Technical Lead - Scala, Python

Dublin, Leinster, Ireland

Job Summary

The Senior Technical Lead at HCL will be responsible for leading technical teams and projects related to Apache Spark, Scala, and Python. The role involves overseeing the design, development, and implementation of scalable and efficient solutions using these technologies.

Key Responsibilities

1. Lead technical teams in the design and implementation of solutions using Apache Spark, Scala, and Python.
2. Provide technical expertise and guidance to team members in resolving complex technical issues.
3. Collaborate with stakeholders to gather requirements and define project scope.
4. Ensure adherence to best practices in coding, testing, and deployment processes.
5. Conduct code reviews and performance optimization activities.
6. Troubleshoot and debug technical issues to ensure seamless project delivery.
7. Stay updated with the latest trends and advancements in Apache Spark, Scala, and Python technologies.
8. Mentor team members and facilitate knowledge sharing within the team.

Skill Requirements

• 8+ years of engineering experience primarily in Data Engineering in an agile production environment.
• High proficiency in using Python or Scala, Spark, Hadoop platforms & tools (Hive, Impala, Oozie, Airflow, NiFi, Scoop), and SQL to build Big Data products & platforms.
• Extensive experience with Databricks, Spark, PySpark or Java + PySpark Processing engine.
• Proficiency in at least one modern programming language such as Python, Java, or Scala.
• Strong computer science fundamentals in object oriented design, data structures, algorithm design, problem solving, and complexity analysis.
• Ability to easily move between business, data management, and technical teams; to quickly intuit the business use case and identify technical solutions.
• Working knowledge in software development engineering paradigms along with data engineering.
• Experience with relational databases as well as NoSQL databases.
• Experience with cloud technologies like Databricks, AWS, or Azure (good to have).
• Basic shell scripting and knowledge of Linux/Unix systems.
• Experience in designing and developing software at scale.
• Strong written and verbal English communication skills.
• Motivation, creativity, self direction, and desire to thrive on small project teams.
• Superior academic record with a degree in Computer Science or related field.

Other Requirements

Relevant certifications in Apache Spark, Scala, or Python are a plus.

Benefits

• Industry benchmarked compensation.
• Best in class healthcare benefits.
• Personal time off.
• Maternity and paternity benefits.
• Access to skills / higher education programs/resources.
• Discounts on products and services via Benefit Box.
• Participation in CSR programs.
• Opportunities to grow and advance your career.

Note: The benefits listed above vary depending on the nature of your employment and the country where you work. Some benefits may be available in some countries but not in all.