Data Engineer (Azure)

Summary

Andersen is hiring a Data Engineer (Azure) to build and support an integration platform for automated data exchange, transformation, and workflow orchestration in a large energy services environment.

The customer is a large energy infrastructure services provider operating in a regulated market. The company focuses on deploying and managing metering and monitoring assets at scale, supporting both residential and commercial users. It delivers installation, maintenance and data-driven energy optimization services, and is expanding into sustainability-oriented solutions aligned with broader efficiency and environmental goals.

The project is focused on developing an integration platform that manages data exchange between external partners and internal systems. It automates file processing, data transformation, and workflow orchestration, while providing business users with clear monitoring and control over incoming and outgoing data streams.

Technology stack: Azure, Synapse Notebooks, Python, Pandas/PySpark, ADLS Gen2, SFTP, REST APIs, CSV/JSON/Parquet, Dataverse APIs.

Responsibilities

• Building and operating file ingestion, validation, transformation and load pipelines in Azure Synapse Notebooks using Python.
• Implementing reliable flows for inbound and outbound files from SFTP to ADLS Gen2, with idempotency, retries and dead-letter handling.
• Producing field mappings and validation rules per file type; handling CSV or JSON or Parquet at scale.
• Integrating with Dataverse and the D365 orchestration layer: writing results, statuses and technical errors according to agreed contracts.
• Instrumenting pipelines with Application Insights and Log Analytics to enable end-to-end correlation and business-visible exceptions.
• Optimizing cost and performance in Azure; following environment strategy DEV / TEST / PROD and CI and CD via Azure DevOps.
• Documenting notebooks, configs and recovery playbooks; supporting UAT together with Integration Engineer and Architect.

Requirements

• Experience as Data Engineer on Azure for 2+ years.
• Strong experience with Python in Synapse Notebooks using Pandas or PySpark.
• Hands-on work with ADLS Gen2, including folder and security model, lifecycle of large file batches.
• Experience integrating and processing data via SFTP, REST APIs, and formats such as CSV, JSON, or Parquet.
• Understanding of reliability patterns, including idempotency, retries, ordering, and backoff.
• Basic understanding of Dataverse APIs and common insert errors.
• Level of English – from Upper-Intermediate and above.

Desired skills

• Azure Functions or Logic Apps experience.
• Observability with App Insights and Log Analytics.
• Awareness of Power Automate triggers used by business users.

Reasons to join us

• Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc..
• The opportunity to change the project and/or develop expertise in an interesting business domain.
• Job conditions – you can work both fully remotely and from the office or can choose a hybrid variant.
• Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee.
• The opportunity to earn up to an additional 1,000 USD per month, depending on the level of expertise, which will be included in the annual bonus, by participating in the company's activities.
• Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated.
• Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies).
• Certification compensation (AWS, PMP, etc).
• Referral program.
• English courses.
• Private health insurance and compensation for sports activities.

Join us!