Azure Data Engineer

InsightglobalIT

InsightglobalIT is on the lookout for a competent Data Engineer who will play a crucial role in crafting scalable data pipelines crucial for effective data processing. The successful candidate will work closely with a cross-functional team to translate data requirements into optimal solutions, ensuring seamless data flow and reliability. The role offers an opportunity to apply your skills in an environment focused on innovative and high-quality data solutions.

In this role you can expect to have the responsibilities:

• Develop and maintain databases and large-scale processing architectures.
• Implement and manage data ingestion, transformation, and storage pipelines.
• Collaborate with teams to understand and address data requirements.
• Monitor data processes to ensure quality and reliability.
• Enhance data pipeline performance through optimization and tuning.

This role comes with the following benefits:

• Opportunity to work with cutting-edge data technologies

This role requires you to have:

• Proficiency in Spark and AWS for data processing.
• Experience with Apache Hive for handling large datasets.
• Understanding of Big Data concepts.
• Database design principles.

You would benefit from having:

• Familiarity with Agile methodologies.
• Shell Scripting for automation.
• Exposure to Informatica for ETL.
• Experience with model training techniques.

Candidates must be able to commute to or relocate to Buffalo, NY 14207 for in-person work.