Principal Data Engineer

Principal Data Engineer

We have an exciting opportunity for an experienced Data Engineer to join our team.

This is a hybrid role, with requirements to work from either our Manchester, Stoke or Peterborough office twice a week.

Role Purpose

The Principal Data Engineer is the senior technical authority for the Databricks Lakehouse platform, responsible for defining engineering standards, architecture, and best practices across the enterprise data ecosystem.

The role leads the design and delivery of scalable, secure, high-performing data products using Databricks, Delta Lake, Unity Catalog and Azure Data Services

The role provides technical leadership, ensuring solutions are robust, cost-effective, highly available, and aligned to business strategy and operational resilience requirements.

Key Accountabilities and Responsibilities

• Being the primary technical contact on data for all new projects, designing the detailed data solution and ensuring that requirements are understood and achievable.
• Design scalable Medallion Architecture solutions (Bronze, Silver, Gold).
• Ensure data governance, lineage, security, and compliance requirements are embedded within all solutions.
• Provide technical leadership across multiple squads.
• Driving continuous improvement through defining and redefining processes within the team, increased automation and providing detailed technical roadmaps of how to achieve our target state.
• Coaching and mentoring other members of the team in technical, delivery and behavioural areas, as well as being the primary escalation point for technical issues and decisions.
• Be the technical contact with vendors and data source supplier to ensure data roadmap is aligned.
• Closely working with stakeholders to ensure they are meeting our needs and delivering to required standards.
• Taking Lead on data product design and development and be the technical expert in all data engineering toolset.
• A heavy focus on resilience, availability and recovery to provide a solid platform that meets Operational Resilience needs for all our systems.

• Proactive technical resource management including availability, capacity, performance and operability.
• Be able to deliver robust data products end to end from requirement gathering to release within time and budget.
• Create clear documentation for the data solution.
• Communicate well with stakeholder and non-technical team.
• To ensure all codes are written are in high standard and perform well.
• Provide technical training and mentor data engineers and support their personal technical development.
• To ensure data dictionaries and models are robust and maintained on a regular basis.

Technical Skill

• Expert-level Databricks experience
• Advanced PySpark development

• Delta Lake architecture and optimisation
• Databricks SQL
• Data warehouse
• Dimensional modelling
• Data Lake
• Databricks Genie
• Python development

Key Skills, Experience and Knowledge

• Extensive Data Engineering experience within enterprise-scale data environments.
• Extensive hands-on experience delivering solutions using Databricks.
• Extensive experience designing and implementing Lakehouse architectures.
• Strong commercial experience using PySpark, Python, SQL and Delta Lake.
• Experience optimising Databricks workloads for performance, scalability and cost efficiency
• Previous experience delivering solutions within insurance, financial services or similarly regulated industries.
• Ability to understand and interpret business needs and how they translate to technical and functional requirements.
• Good project management skills; experience managing projects / requests with multiple senior stakeholders.
• Experience managing and mentoring engineering teams.

About Us

Markerstudy Group Services Limited is one of the largest Managing General Agents in the UK. With a strong presence in the UK motor insurance market, we specialise in niche motor cover, where our solid market knowledge and experience enables us to create highly targeted products.

Our success is underpinned by our underwriting strategy to identify and apply special risk factors to the customers’ advantage. That, and our skilled underwriting technicians who are friendly, accessible and empowered to make decisions.

Benefits

• Starting with 25 days annual leave which increases with service

• Opportunity for a yearly bonus
• Private Medical Insurance
• Life Assurance 4x annual salary
• Collaborative, fast paced working environment
• Hybrid Working
• Modern, vibrant offices