Big Data Engineer

Key Responsibilities

End-to-End Project Ownership

• Gather requirements directly from business users.
• Perform business and data analysis.
• Design system architecture.
• Develop solutions.
• Support testing and validation.
• Work with the DevOps team on deployment.
• Provide ongoing support after release.

AI/GenAI Solution Development

• Build applications leveraging LLMs, such as OpenAI APIs.
• Implement use cases including content summarisation, intelligent filtering, and categorisation.
• Evaluate when AI is appropriate versus traditional coding approaches.

Full Stack Development

• Work across the entire stack rather than in specialised roles.
• Handle backend logic, data processing, and integration layers.
• Work without separation into traditional DB, UI, and middleware roles.

Architecture & Design Decision-Making

• Independently design and justify technical architecture.
• Make decisions for greenfield projects.
• Explain trade-offs and reasoning behind implementation choices.

Stakeholder Interaction

• Engage directly with business users and the product owner.
• Translate business needs into technical solutions.
• Handle feedback and change requests directly.

Agile & Independent Working Model

• Work in a small, lean pod.
• Handle projects independently with minimal supervision.
• Participate in design discussions and own individual project delivery.

Technology Stack (Indicative)

• Primary language: Python.
• AI/ML: LLM integrations, including OpenAI or similar.
• Data platforms: Databricks and Delta Lake.
• Cloud: Azure ecosystem.
• Visualisation: Power BI (optional; not mandatory).

Required Skills & Capabilities

Core Skills

• Strong full-stack development capability.
• Solid understanding of system architecture and design.
• Experience building end-to-end applications.

AI & Data Skills

• Experience working with GenAI and LLM-based solutions.
• Ability to integrate AI into real-world use cases.

Technical Skills

• Programming: Scala, Java, Spark.
• Tools: GitLab, Sonar, Docker Hub, container registry, Nexus, Postman, Azure CLI.
• Build scripting: Gradle, Maven, Shell.
• Packaging: Shell.
• Cloud: Azure.
• Hosting: container-management platforms, including Kubernetes and Databricks.
• Data: Storage Accounts, PostgreSQL, Spark, and Databricks API knowledge.
• Security: Azure AD, SAML token-based authentication, single sign-on, HashiCorp Vault, Managed Identities, and Service Principals.
• Monitoring: Log Analytics, Azure Metrics, and Azure Monitor.
• Infrastructure as Code: ARM.

Test Automation

• Test frameworks: Cucumber, JUnit, Mockito.
• Tools: GitLab, Sonar, Docker Hub, container registry, Nexus, Postman.
• Build scripting: Maven and Shell.
• Service/API testing: thoroughly test endpoints.
• End-to-end testing of integrated systems, supported by continuous integration.
• Solid knowledge of QA methodologies, test planning, system dependencies, and product integration phases.