Python AI with Azure

Python AI Engineer (Azure Document Intelligence + Feedback-driven Retraining/Improvement)

Role summary

• Own the document ingestion AI pipeline  using Azure Document Intelligence  and implement a feedback loop from the “Extract & Audit” UI to continuously improve extraction quality.

Key responsibilities

• Build ingestion pipeline that supports:
  • Multiple file inputs  + optional text input
  • Template-driven extraction (selected “Extraction & Audit Template”)
• Implement extraction outputs aligned to UI:
  • Produce Tags  (name, extracted value, source file)
  • Produce Audit Rules  with pass/fail + “Audit Notes”
  • Provide Preview payloads  for document/text/email content and per-rule preview
• Build human-in-the-loop feedback capture :
  • Capture user decisions: Accept/Reject/Un-map/Manual Map/Manual Audit
  • Persist feedback with traceability: doc_id, page/region, tag_id, old_value, corrected_value, user, timestamp
• Implement continuous improvement/retraining strategy (depending on approach):
  • Azure DI custom models training  (labeling + training sets + model versions) and redeploy model versions safely
  • Rule/prompt improvements for post-processing normalization (dates, currency, identifiers)
  • Build evaluation harness: precision/recall for tags + audit rule pass-rate tracking
• Enable export outputs:
  • Generate structured outputs for FundPanel CSV, Xero CSV, Email Body  and “Export ALL” 

Technical requirements

• Strong Python (FastAPI preferred), async patterns, typing, clean architecture
• Hands-on with Azure Document Intelligence :
  • Prebuilt vs custom models, training datasets, model lifecycle & versioning
  • Confidence scores & field extraction best practices
• Data engineering essentials:
  • Blob storage integration (staging/finished), metadata storage (Postgres/Cosmos)
  • Batch processing + retry + idempotency (avoid duplicate extraction)
• ML/quality loop:
  • Designing feedback datasets, sampling strategies, active learning concepts
  • Offline evaluation pipeline + dashboards (quality, drift, rule pass rates)
• Operational readiness:
  • Logging, tracing, metrics; per-document cost tracking
  • Secure handling of sensitive docs (PII redaction optional depending on use case)

Nice-to-have

• Azure AI Search indexing (if you also store extracted text for retrieval)
• Experience with document parsing libraries (PyMuPDF/pdfplumber) alongside DI
• Prompting + post-processing (LLM-assisted normalization) with strict schema output