Data Engineer (Remote)

Key Responsibilities:

Data Ingestion:

• Design and implement pipelines to ingest structured and...
  Login to continue

  Data Engineer | AI Dev Lab | Colombia

  Key Responsibilities:

  Data Ingestion:

  • Design and implement pipelines to ingest structured and unstructured data from various sources (APIs, web scraping, databases, file systems, streaming platforms, etc.).
  • Ensure reliability, scalability, and efficiency of data ingestion processes.
  • Standardize data across highly varied data sources

  Database Design and Management:

  • Set up and manage relational (SQL) and non-relational (NoSQL) databases, ensuring optimal performance.
  • Develop schemas, indexes, and views to support data storage and retrieval.
  • Optimize databases for storage and query efficiency.

  Infrastructure and Server Setup:

  • Configure servers for data storage, processing, and analytics.
  • Implement user permissions, access control, and other security measures.
  • Automate deployment and scaling of server resources.

  Data Security and Compliance:

  • Ensure data encryption in transit and at rest.
  • Implement robust authentication and authorization frameworks.
  • Monitor systems for vulnerabilities and ensure compliance with data privacy laws.

  Integration with Processing Pipelines:

  • Connect data storage systems to processing tools (e.g., Python, Spark, ETL frameworks).
  • Work with data scientists, analysts, and other teams to integrate and operationalize data workflows.
  • Build and manage data transformation layers for analytics and reporting.

  Production Environment Management:

  • Design and implement production-ready environments with high availability and fault tolerance.
  • Monitor data pipelines and storage systems to ensure uptime and performance.
  • Automate workflows using tools like Apache Airflow, Luigi, or similar orchestration frameworks.

  Required Skills and Qualifications:

  Education:

  • Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent experience).

  Technical Skills:

  • High proficiency in database management systems: SQL (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra, DynamoDB).
  • Experience with data pipeline tools: Apache Kafka, Apache Nifi, or AWS Glue.
  • Competent with cloud platforms (AWS, Azure, GCP) and containerization (Docker, Kubernetes).
  • Excellent programming skills in Python, Java, or Scala.
  • High proficiency with ETL tools and frameworks.
  • Strong experience with CI/CD pipelines for deploying data applications.
  • Knowledge of machine learning workflows and integrating with ML models.

  Security Knowledge:

  • Understanding of data encryption techniques.
  • Familiarity with IAM (Identity and Access Management) systems.
  • Experience implementing secure data environments.

  Soft Skills:

  • Strong problem-solving abilities and attention to detail.
  • Excellent English communication skills to collaborate with cross-functional teams.
  • Ability to manage multiple projects and prioritize tasks effectively.

  Preferred Qualifications:

  • Experience with big data technologies (Hadoop, Spark).
  • Certification in cloud technologies (AWS Certified Data Analytics, Google Cloud Professional Data Engineer).
  • Minimum of 5 years of work experience.
    

  Show more

  Show less

  Tagged as: remote, remote job, virtual, Virtual Job, virtual position, Work at Home, work from home