Problem
Finding reward flights on Qantas and Velocity is slow. I wanted a fast, utilitarian tool to scrape availability directly and bypass clunky airline interfaces.
Approach
The app uses a Next.js frontend backed by PostgreSQL, Prisma, and Redis. The heavy lifting happens in a separate Node.js worker running Playwright and Chromium.
It is deployed on AWS using Terraform and runs on a Graviton2 ARM64 instance. CI/CD pushes multi platform Docker images to ECR, and Docker Compose handles orchestration through AWS SSM.
Technical decisions
Scraping airlines headless means dealing with anti bot systems, transient network drops, and page state changing underneath the worker.
For Velocity, I guarded page.evaluate calls against SPA navigations, handled transient Chromium timeouts, and managed browser user agents to reduce Imperva blocks. For Qantas, I built region mode fanout that respects rate limits by serializing source calls. When broad month view lookups started returning 403s, I added fallback logic to pivot into single page, day by day probes.
Outcome
Live scraping is stable, and the next step is expanding the background worker to support saved searches and email alerts.