Job Title: Azure Data Engineer
Location: Hopkins, MN
Duration: 6+ Months
Project Summary:
This project involves building AC360 data pipelines in Azure using Spark technologies. As part of this initiative, the development of Zafin data pipelines in Azure is also included.
Key Responsibilities:
1. Identify Requirements and Develop Solutions
· Identify requirements for RMS replacement and develop solutions in Azure.
· Re-engineer solutions to retrofit the rules with AC360 in Azure.
2. Data Pipeline Management
· Register data pipelines in Azure, onboard datasets, and enable data processing.
· Build Airflow DAGs and ensure scheduling is integrated.
3. DevOps and Production Management
· Apply a DevOps mindset, take ownership of production success, and optimize operations via automation, active alerting, and self-healing.
· Lead the resolution of production issues to ensure high availability and performance.
4. Code Development and Review
· Develop high-quality code and define best engineering practices.
· Perform peer code reviews to ensure successful deliverables with engineering excellence.
5. Documentation
· Document engineering artifacts such as technical design documents, flowcharts, system context diagrams, and code comments.
6. Collaboration
· Collaborate cross-functionally with product owners, data scientists, business users, project managers, and other engineers to achieve elegant solutions.
· Work with other engineers on the team to elevate technology and consistently apply best practices.
7. End-to-End Development and Testing
· Execute true end-to-end development and testing of software products in an agile lifecycle.
· Partner with key technical and business stakeholders to ensure the quality of the final product through manual and automated tests.
· Support the new software as it is moved to production.
Required Skills/Technologies:
· Mainframe COBOL: Good knowledge to understand the rules in Strata rules engine.
· Spark-Scala or Spark-SQL, Spark-Py: Minimum 5 years of experience.
· Airflow, Elastic, Azure: 3 years or higher.
· Big Data Management: 10 years or higher.
· Microservices-Based Applications: 5+ years.
· Azure, Kubernetes, PostGres: Experience required.
· Maven, Docker, Kubernetes, Kibana, ELK, Jenkins, Spring Boot: 4+ years.
· CI/CD Processes (Maven, Git, Jenkins): 4+ years.
· Cloud Technologies: 4+ years.
· Server-Side Languages (Python, Java, GoLang): 3+ years.
· Communication Skills: Excellent verbal and written communication skills, 10 years or higher.
· Technical, Logical, Analytical, and Problem-Solving Skills: 10 years or higher.
Preferred Skills/Attributes:
· Experience with Spark, Scala, Python, Azure stack.
Required Skills : • Mainframe COBOL: Good knowledge to understand the rules in Strata rules engine. • Spark-Scala or Spark-SQL, Spark-Py: Minimum 5 years of experience. • Airflow, Elastic, Azure: 3 years or higher. • Big Data Management: 10 years or higher. • Microservices-Based Applications: 5+ years. • Azure, Kubernetes, PostGres: Experience required. • Maven, Docker, Kubernetes, Kibana, ELK, Jenkins, Spring Boot: 4+ years. • CI/CD Processes (Maven, Git, Jenkins): 4+ years. • Cloud Technologies: 4+ years. • Server-Side Languages (Python, Java, GoLang): 3+ years. • Communication Skills: Excellent verbal and written communication skills, 10 years or higher. • Technical, Logical, Analytical, and Problem-Solving Skills: 10 years or higher.