Job Description
Position Overview:
As a Data Platform Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines, data processing systems, and data integration solutions using AWS, Snowflake, and Kafka. You will work closely with cross-functional teams, including data scientists, analysts, and software engineers, to ensure that our data platform is robust, efficient, and meets the needs of the business.
Key Responsibilities:
- Data Infrastructure Management:
Design, implement, and optimize cloud-based data infrastructure on AWS. Ensure high availability, scalability, and reliability of the data platform. - Data Pipeline Development:
Build and manage ETL pipelines to move and process data efficiently using AWS services like Lambda, S3, Redshift, Glue, and Kafka. Ensure smooth data ingestion and transformation workflows. - Snowflake Integration & Management:
Develop, optimize, and maintain Snowflake data models and schemas. Work on data integration, performance tuning, and monitoring to ensure that Snowflake is operating at optimal performance. - Real-time Data Streaming:
Design and implement real-time data streaming solutions using Kafka for efficient data ingestion and processing. Develop microservices for event-driven architectures. - Automation & Monitoring:
Implement automated data workflows and deploy monitoring tools for real-time visibility into data quality, pipeline performance, and system health. - Collaboration:
Work closely with data scientists, analysts, and software engineers to understand data requirements and translate them into technical solutions. Collaborate to deliver data-driven products and insights. - Best Practices & Security:
Ensure data security, privacy, and compliance by implementing best practices in data governance, access controls, and encryption. Stay up to date with industry standards and emerging technologies.
Skills & Qualifications:
- Required Experience:
- 3+ years of experience in building and maintaining cloud-based data platforms, pipelines, and systems.
- Strong expertise in AWS services, including Lambda, S3, Glue, Redshift, and CloudWatch.
- Hands-on experience with Snowflake, including data model development, performance tuning, and managing ETL workflows.
- Solid experience with Apache Kafka for data streaming, message brokers, and real-time data processing.
- Proficiency in scripting languages (e.g., Python, SQL, or similar).
- Experience with CI/CD practices and tools for deploying data infrastructure.
- Preferred Skills:
- Experience with other cloud providers (Azure, GCP) is a plus.
- Familiarity with data orchestration tools like Apache Airflow or AWS Step Functions.
- Experience with containerization (e.g., Docker, Kubernetes) and serverless architecture.
- Understanding of data warehousing concepts, including OLAP and OLTP systems.
- Education:
- Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field (or equivalent work experience).
