As a Senior Big Data Engineer, you will be responsible for designing, building, and maintaining scalable and robust data processing pipelines and infrastructure for storing, processing, and analyzing large volumes of data. You will work closely with data scientists, data engineers, and business stakeholders to architect data solutions that drive insights and value for the organization.
Professional Skills
Hadoop
Spark
Kafka
HBase
SageMaker
TensorFlow
PyTorch
Job Responsibilities
Design, develop, and maintain end-to-end data pipelines for ingesting, processing, and transforming structured and unstructured data from various sources using technologies like Apache Hadoop, Apache Spark, and Apache Kafka.
Implement and optimize distributed data processing workflows for batch and real-time data ingestion, leveraging frameworks like Apache Flink or Apache Beam for stream processing and complex event processing (CEP).
Build and manage scalable data storage solutions, including data lakes, data warehouses, and NoSQL databases like Apache HBase, Apache Cassandra, or MongoDB, ensuring data accessibility, reliability, and security.
Collaborate with data scientists and analysts to implement machine learning pipelines for model training, evaluation, and deployment, integrating with machine learning frameworks like TensorFlow, PyTorch, or scikit-learn.
Ensure data quality and governance by implementing data validation, cleansing, and enrichment processes, leveraging tools like Apache NiFi, Apache Atlas, or Informatica for data lineage and metadata management.
Optimize data infrastructure and performance, tuning database queries, optimizing data partitioning and indexing strategies, and troubleshooting performance bottlenecks to meet scalability and performance requirements.
Implement and maintain data security and compliance measures, including data encryption, access control, and auditing, to protect sensitive data and ensure regulatory compliance.
Provide technical leadership and mentorship to junior members of the Big Data team, fostering a culture of innovation, collaboration, and continuous learning.
Job Qualifications
Bachelor's or Master's degree in computer science, engineering, or a related field.
Proven experience in designing and building scalable data processing pipelines and infrastructure in production environments, preferably using distributed computing technologies like Apache Hadoop or Apache Spark.
Proficiency in programming languages such as Java, Scala, or Python, with experience in developing and optimizing data processing applications and algorithms.
Strong understanding of big data concepts and technologies, including data modeling, storage, processing, and analysis, with hands-on experience in working with large-scale datasets.
Experience with cloud-based big data platforms and services such as Amazon EMR, Google Cloud Dataproc, or Microsoft Azure HDInsight.
Familiarity with data integration and ETL tools like Apache NiFi, Talend, or Informatica for orchestrating data workflows and integrating data from multiple sources.
Excellent communication and interpersonal skills, with the ability to effectively collaborate with cross-functional teams and stakeholders.
Soft Skills
Strong problem-solving abilities and attention to detail, with a proactive and results-oriented mindset.
Effective time management and organizational skills, with the ability to prioritize tasks and meet deadlines in a fast-paced environment.
Adaptability and willingness to learn new technologies and methodologies, staying updated with the latest trends and best practices in big data engineering.
Strong teamwork and collaboration skills, with the ability to work independently as well as part of a multidisciplinary team.
Looking for none Technical jobs instead? Apply here
Why AngelsCloud
Joining AngelsCloud opens the door to a vibrant community of tech enthusiasts, offering exciting projects, continuous learning opportunities, and the chance to make a meaningful impact in the dynamic world of IT.
Great Place to Work
This accolade stems from our workers' feedback, driving us to constantly innovate and implement fresh ideas.
Many Fascinating Projects
Work with exalted high-ranking brands from across the globe on short or long-term contracts, ensuring a rich and diverse portfolio.
Projects Suited to You
Whether you're looking to change your career path or your current project is coming to an end, we'll find you a new one! Our team has a plan for this.
The Best Place for Personal Development
Every project involves world-class Principal or Architect Experts, providing ample opportunities for continuous learning and personal development.
Earn Your Own Shares, Multiply Your Profit
Our special program allows everyone who significantly impacts any project, customer, or company development to gain profit through various shares. Additionally, referencing new experts for any new customer may create an additional permanent source of income.
Many Experts from Different Locations
Our team consists of many experts from different locations, bringing diverse perspectives and expertise to every project.
Employment Stability
We provide employment stability to our team members, ensuring consistency and security in their careers.
Open-minded and Available Management Team
Our management team is always approachable, encouraging open dialogue to ensure your ideas and needs are heard and addressed.
True Values
Our values are not just empty words - we are guided by them every day in our work and interactions.