Job Title: Data Engineer
Location: San Francisco, CA
Duration: 6+ Months
Contract: W2
Job Summary:
- Continuously improve our data infrastructure and stay ahead of technology.
- Build strong cross-functional partnerships with Data Scientists, Analysts, Product Managers and Software Engineers to understand data needs and deliver on those needs.
- Evaluate and select appropriate BI tools and technologies based on the organization's needs. Run POC and the implementation and configuration of BI software, including data visualization tools, reporting platforms, and analytics solutions
- Stay updated with the latest trends, technologies, and best practices in the BI field. They explore opportunities for process improvement, automation, and innovation to enhance the organization's data analytics capabilities
- Establish Data governance policies and procedures to ensure data quality, consistency, and security. Define access controls, data privacy measures, and data retention policies to comply with regulatory requirements and protect sensitive information
We’re excited about you if you have:
- 8 -10 years of experience in data engineering and designing scalable ETL pipelines
- Advanced Proficiency in Hadoop, Scala, Spark, Kafka, and SQL
- 6+ years of experience with cloud platforms like Azure, AWS, Google
- Be skilled at architecting large and complex data pipelines.
- Experience performing data analysis and data exploration.
- Experience working with Cloud technologies, Azure, Databricks, ADLS, Spark, Cosmos DB, and other big data technologies.
- Experience working in an agile delivery environment.
- Experience working in multi-developer environments, using version control.
- Excellent knowledge of data structures and design patterns?
- Preferred experience with data integration tools
- Work with cross-functional agile teams to drive projects through full development cycle.
- Help the team improve with the usage of data engineering best practices.
- Write SQL for processing raw data, data validation, and QA
- Knowledge working with APIs to collect or ingest data
- Strong Database knowledge, SQL & No-SQL preferred
- Communication Skills Data Engineers are part of a team, working with database administrators, data analysts and management, and need to be effective communicators.
- Problem-Solving Skills Data Engineers look at an issue that needs to be solved and come up with solutions quickly.
- Experience integrating and building data platforms in support of BI, Analytics, Data Science, and real-time applications.
- Strong communication skills, with the ability to initiate and drive projects proactively and accurately
- Experience with Datastage or other ETL tools, experience with reporting tools like Cognos, SSAS Cubes, Tableau, Thoughtspot.
Required Skills : SQL, Python, Spark or PySpark, and Scala. Databricks. ETL Tools, preferably Datastage, but others like FiveTran or Informatica are ok. BI Reporting Tools – SASS Cubes, Tableau, ThoughtSpot, Power BI, etc. Platform Migration experience. Strong communication and cross-functional support skills.