大數據處理工程師

Job description

Responsibilities

Job Summary/Role: The Big data developer team works closely with the manufacturing functions in all relative data fields, such as, 1) connecting and automating the data pipeline from MFG file servers, API, RDB, etc., 2) building structurally data warehouse and data market for flexible application or dashboard, and 3) also handling data cleansing and data governance. In order to accelerate the development of application/analysis, Big data developer will be the individual contributor with strong experience in the technologies of Big Data, Data warehouse, SQL and NoSQL etc.

Also Big data developer will also work closely with data analysts/data scientists to solve the real MFG cases.

Responsibility

• Collaborate with users, data analysts, data scientists to develop flowcharts, layouts and documents to identify solutions.
• Architect on real MFG needs and translate requirements into technical needs.
• Identify and understand the data sources that are relevant for analysis, and implement efficient data engineering functions (including data ingestion, cleaning, ETL, integration, etc.)
• Consolidate the requirements and using data engineering skills to design a robust Data Mart.
• Participate in design, architecture review and deployment of Big data and data warehouse solutions.
• Design for automation and monitoring mechanism in a continuous integration environment.
• Bring new data technology with POC to solve existing data challenges.
• With high performance and comprehensive SQL to process data in relational databases (or Map/Reduce style querying for NoSQL databases).
• Work closely with user/IT/data science team to design and develop scalable data pipeline or end-to-end analytical workflow.
• Provide direction, training, and guidance to less experienced team members.

Requirements

Qualifications:

• Proficient in R/Python and related data processing packages (e.g., Pandas)
• Proven experience in developing data pipeline, data processing workflow, or data analytics system
• Master in using SQL to process data in relational databases such as Oracle and PostgreSQL as well as NoSQL databases.
• Ability to thrive in a dynamic, collaborative team environment
• Ability to document requirements and specifications.
• Excellent verbal and written communication skills.
• (Preferred) Have experience in data mining and machine learning
• (Preferred) Familiar to build up message queue environment and use it.  (Prefered) Have experience working with Hadoop ecosystem (e.g., Map Reduce, Hive, Oozie) or Spark, Kafka, Flume, etc.  (Plus) Familiar with Linux operation system  (Plus) Have experience in system/database performance tuning.