Aws Opensearch Engineer (Remote)

We are seeking an experienced AWS OpenSearch Engineer to join our clients team on a long term contract basis. This critical role will focus on optimizing and enhancing the existing AWS OpenSearch environment, which is integral to monitoring and analyzing the clients High-Performance Computing (HPC) environment. The ideal candidate will be responsible for optimizing performance, incorporating new data sources, and developing advanced queries to support the analysis and management of the HPC environment.

Time Zone: CST

Responsibilities:

Environment Deep Dive and Optimization: Gain a comprehensive understanding of the current AWS OpenSearch implementation, including architecture, data sources, and indexing strategies. Implement performance tuning measures to improve the efficiency and reliability of the environment. HPC Performance Analysis: Develop and refine OpenSearch queries to monitor job run times, detect long-running or stalled jobs, and identify performance bottlenecks within the HPC environment. FinOps Integration: Work with existing FinOps data within OpenSearch to correlate job run times with associated costs. Develop dashboards and visualizations to track performance, identify optimization opportunities, and improve cost-efficiency in the HPC environment. Data Ingestion and ETL: Integrate data from multiple sources, including AWS CloudWatch and other monitoring tools, into the OpenSearch environment. Develop and implement data ingestion pipelines for new data sources as needed. Reporting and Visualizations: Create dashboards and reports to track system performance, job costs, and other key metrics that help proactively identify issues and optimization opportunities.

Qualifications:

Experience: Minimum of 3 years of hands-on experience with AWS OpenSearch or Elastic search. Technical Expertise: Strong understanding of OpenSearch architecture, indexing principles, and the development of queries using DSL (Domain-Specific Language) or SQL. Data Management: Experience in data ingestion, ETL processes, and performance tuning of OpenSearch clusters. AWS Services: Familiarity with AWS services relevant to OpenSearch, including EC2, S3, IAM, and CloudWatch. Problem-Solving: Excellent problem-solving and analytical skills, particularly in performance optimization and troubleshooting. HPC Knowledge: Experience working within High-Performance Computing (HPC) environments and understanding of job scheduling systems. Preferred Qualifications:

FinOps Expertise: Experience with FinOps concepts and cost optimization strategies, particularly within AWS environments. DevOps Practices: Familiarity with DevOps tools and practices, including infrastructure automation and continuous integration/continuous delivery (CI/CD) pipelines. Scripting Skills: Proficiency in scripting languages such as Python and Bash, with experience in automation and data manipulation. Visualization Tools: Experience with data visualization tools such as Kibana, used for creating dashboards and monitoring performance metrics.