# Muhammad Sufiyan Baig — Lead Software Engineer

> Backend engineer who builds AI-integrated production systems — and is moving toward MLOps.

Muhammad Sufiyan Baig is a Lead Software Engineer with 2.5+ years building production backends and AI-integrated systems in FastAPI, Next.js, TypeScript and PostgreSQL. 20+ shipped projects across fintech, SaaS and multi-tenant platforms, and moving toward MLOps. Open to backend and full-stack roles.

- 2.5+ years of professional experience (since September 2023)
- 20+ shipped projects
- Based in Pakistan (PKT, UTC+5), available for remote work worldwide
- Studying BS Data Science at SMIU, graduating 2027

This file contains the complete portfolio content: every project as a full
case study, plus experience and skills. The short index lives at /llms.txt.

---

# Experience

## Full Stack Developer and Project Manager — Element 2.0158 UG & Co KG

July 2025 - Present · Stuttgart, Germany (Remote)

Engineered full backend for an international dart training app (Stuttgart, Germany): 31 REST API endpoints covering auth, game sessions, training plans, and analytics; JWT dual-token auth, OAuth 2.0 (Google and Apple), and RevenueCat subscription integration. Designed 30+ database tables with Drizzle ORM; implemented X01 game engine, timezone-aware training schedules, score recommendations, dashboard analytics, and One Signal push notifications

## Lead Software Engineer — Innovative Widget

Sep 2024 – Present

Mentored and led junior developers to encourage collaboration and growth. Designed, developed and deployed user-friendly full-stack web applications.

## Full-Stack Developer — Panacloud Pvt Ltd

Sep 2023 – Sep 2024

Developed and managed responsive websites using Frameworks & Libraries. Collaborated with teams to improve website performance and user experience.

---

# Projects

## CryptoFleet.io

URL: https://muhammadsufiyanbaig.vercel.app/projects/client/cryptofleet
Category: client (featured)
Built for: Panacloud
Links: Live: https://cryptofleet.io

A full-featured cryptocurrency trading platform supporting P2P and spot trading, built with a serverless architecture and real-time market data integrations.

### Problem

Retail crypto beginners in Pakistan and similar markets wanted to practice trading without risking real capital, but existing exchanges offered no safety net — a wrong trade meant real losses, and there was no structured way to learn while doing. Separately, buying/selling crypto for fiat outside of centralized exchanges usually meant trusting an unverified stranger on Telegram or WhatsApp groups, with no escrow and no dispute process. Beginners had no single place to learn the fundamentals, test their skills in a challenge, and then trade P2P with a safety mechanism.

### Overview

CryptoFleet.io is a full-featured cryptocurrency trading platform supporting P2P and spot trading, built with a serverless architecture and real-time market data integrations from Binance, CoinMarketCap, and CoinGecko. The platform includes real-time candlestick charts, multi-coin wallet management, gamified trading competitions, and a referral system — all secured with AWS Cognito authentication. The admin dashboard allows managing challenges, participants, orders, referrals, and rewards. A Learning Center with educational courses and completion badges rounds out the platform's features.

### Features

- P2P Trading — Create buy/sell ads, browse marketplace, manage active trades and order history
- Spot Trading — Real-time candlestick charts powered by Binance API via Lightweight Charts
- Wallet Management — Multi-coin wallet with balance tracking
- Trading Challenges — Gamified trading competitions with leaderboards and rewards
- Referral System — Referral tracking dashboard with reward claiming
- Learning Center — Educational courses with completion badges
- Admin Dashboard — Manage challenges, participants, orders, referrals, and rewards
- Push Notifications — Real-time alerts for trade updates
- Dark / Light Mode — Full theme switching support

### Stack

Next.js 14, TypeScript, Tailwind CSS, Redux Toolkit, AWS Cognito, AWS AppSync, GraphQL, Binance API, CoinMarketCap, Vercel

## Square One Orbit

URL: https://muhammadsufiyanbaig.vercel.app/projects/client/square-one-orbit
Category: client (featured)
Built for: Innovative Widget
Timeline: Apr 2026 – Jul 2026 (3 months, ongoing)
Role: I was the primary backend engineer and owned the API end to end: schema and migrations, authentication, the six-role RBAC model, the full ticket lifecycle, push/email notifications, the audit trail, scheduled jobs, and the Vercel deployment. That is 56 endpoints across 11 routers and 9 Postgres tables, plus the Alembic migration chain. A second engineer (Zaeem) contributed filtering enhancements that I reviewed and merged; everything else in the history under the core domain, auth, and infra is mine.
Links: Live: https://play.google.com/store/apps/details?id=com.innovativewidget.squareone

Full-stack task and ticketing platform for a 24/7 shopping mall — 102 employees across 9 departments, 6-tier RBAC, camera-only proof of completion, subticket dependencies, automated monthly performance reports, and a FastAPI backend with 44 endpoints.

### Problem

Square One Mall ran maintenance and inter-department requests on WhatsApp messages and paper — a request for "AC not cooling on Level 3" had no owner, no deadline, no proof it was ever done, and no record of who dropped it. Managers could not see what was outstanding across departments, staff on the floor had no structured way to claim or hand off work, and there was no accountability once a job was marked done. Everyone was coping with screenshots and memory.

### Constraints

The people using it are floor staff and department managers, not office workers — much of the work happens on phones, frequently in basements and service corridors with no signal, so the mobile app had to work offline and sync later. The backend runs on Vercel's serverless platform (Hobby tier), which rules out long-lived processes and has a hard ceiling on cron frequency. Proof of completion had to be real — a photo or video, not a checkbox — which pulled object storage and upload limits into scope. And the role structure was fixed by the org chart: super admin, admin, head of department, department manager, supervisor, employee, each seeing a different slice of the system.

### Architecture

The system is a stateless FastAPI service in front of a single Postgres database
(Neon), deployed as a Vercel serverless function. Statelessness was not a
preference — it was forced by the platform, and it shaped everything downstream.
The obvious first design used APScheduler for recurring jobs and FastAPI
`BackgroundTasks` for sending email after a response; both silently do nothing on
serverless because the process is frozen or killed the moment the HTTP response is
returned. So scheduled work (deadline alerts, monthly reports, archival) was moved
to dedicated cron endpoints triggered by Vercel Cron with a shared secret, and the
in-process APScheduler is kept only for local/non-serverless runs, gated on the
`VERCEL` env var.

RBAC lives in one module (`permissions.py`) as pure functions over the user and
the target row, rather than being scattered through the route handlers. Every
mutation calls the same `can_*` / `is_admin_equivalent` gate, so the hierarchy has
exactly one definition and the "management department behaves like admin" override
is expressed once. Notifications and the audit trail are separate services because
they are cross-cutting — nearly every state change writes an audit row and fans out
a notification, and I did not want that logic copy-pasted into a dozen endpoints.
The audit log is append-only with JSONB before/after snapshots so history can never
be edited through the API.

```flow
Mobile app :: Offline-first client; queues writes, replays on reconnect with a client_reference_id idempotency key
API layer :: FastAPI on Vercel — JWT (python-jose/bcrypt) auth, in-memory per-IP rate limiting, Pydantic v2 validation
RBAC gate :: permissions.py — one hierarchy definition, checked before every mutation
Tickets router :: Lifecycle orchestration: pending → acknowledged → in_progress → completed → resolved; idempotent create; self-claim on acknowledge
Services :: notification_service (Firebase FCM), audit_service (append-only JSONB), storage_service (S3/R2 proof upload)
Postgres (Neon) :: 9 tables; unique (generator_id, client_reference_id) enforces dedup at the DB, not the app
```

### Challenges

- Serverless silently ate background work: The first version sent report emails via FastAPI `BackgroundTasks` and scheduled alerts via APScheduler. Neither errored — they just never ran, because Vercel freezes the function after the response. The fix was to stop relying on process lifetime entirely: move every recurring job to an explicit cron endpoint invoked by Vercel Cron, and do the email send inline before returning. "It works locally" was the whole trap.
- FCM push failed three separate ways: Push dispatch got repaired across several commits — a blocking `firebase-admin` call sitting inside an async path, a malformed data payload the client couldn't parse, and failures that were swallowed so nothing surfaced. What finally worked was treating push as best-effort and isolated: it never blocks or fails the request that triggered it, and recipient lists are de-duplicated before send so one user can't get the same alert twice.
- Audit writes 500'd on JSONB: Logging a ticket update crashed because the before/after snapshots contained `datetime` and `UUID` objects, which psycopg can't serialize into a JSONB column. I wrote a small recursive `serialize_value` that walks the snapshot and normalizes UUIDs, datetimes, enums, and nested dict/lists into JSON-safe values before the write.
- Offline sync produced duplicates, then 500s: When the mobile app replayed a queued create after reconnect, a lost response caused a retry that inserted a second ticket. I added a client-supplied `client_reference_id` with a unique `(generator_id, client_reference_id)` constraint so the *database* rejects the duplicate, and the handler returns the existing row instead of erroring — the guarantee holds even for two concurrent retries. A follow-up bug was subtler: the offline queue re-serialized timestamps as timezone-aware (`+05:00` / `Z`), and subtracting them from a naive `utcnow()` raised `TypeError` and 500'd on every retry while the same body worked online. I reproduced it with a 14-variant payload matrix against the real failing parent ticket, then normalized every inbound datetime to naive UTC at the boundary and mapped stale foreign-key references to a clean 422 instead of a 500.
- Self-claim under concurrency: An unassigned, department-routed ticket has to be claimable by any eligible floor worker via acknowledge — but exactly one person can win. A read-then-write check would race. I made the transition an atomic conditional update (`WHERE status = 'pending'`) so the status flip is the lock; the winner gets the assignment, the loser gets a 409.

### New ground covered

- Serverless FastAPI on Vercel: I had built FastAPI on always-on servers before, never on a platform that kills the process after each response. I now understand that anything depending on process lifetime — schedulers, background tasks, in-memory state, warm caches — is a liability there, and that "no error" is not the same as "it ran."
- Server-side Firebase Cloud Messaging: First time integrating `firebase-admin` for push, including shipping service-account credentials to a serverless host as base64 rather than a file. I learned how much of push reliability is about failure isolation and payload shape, not the send call itself.
- Offline-first idempotency design: I had not designed for a client that replays writes. The lasting lesson is to push the uniqueness guarantee down to a database constraint — an idempotency key with a unique index survives concurrency and process restarts in a way an application-level "check if it exists first" never can.

### Takeaways

- On serverless, silent no-ops are the dangerous failure mode; anything that relies on the process staying alive has to be redesigned, not ported.
- Idempotency belongs at the database boundary — a unique constraint on a client reference id beats any application-level duplicate check under retries and races.
- Timezone bugs hide until a second client serializes datetimes differently; normalizing to naive UTC at the API boundary is cheaper than chasing them later.
- A single-definition RBAC module is worth the up-front discipline — one hierarchy function is auditable in a way that per-endpoint checks never are.
- Distinguishing a client-data error (stale reference → 422) from a server fault (500) is what stops a mobile sync queue from retrying forever on a bad row.

### Journey

I started this treating Vercel like any Linux box with a Python process on it. I wired up APScheduler for the recurring jobs and used FastAPI's `BackgroundTasks` to send report emails after returning a response, tested it locally, and moved on. It all worked on my machine and quietly did nothing in production. Finding that out — that there were no errors, no logs, just reports that never arrived — was the first real humbling of the project and the moment I stopped trusting "it works locally" on serverless. Push notifications were the next long slog. FCM failed in three different ways over a couple of weeks — a blocking call in the wrong place, a payload the app couldn't read, and errors that were being swallowed — and each looked like the previous fix hadn't worked. The thing that clicked was to stop treating push as part of the request and treat it as a side effect that is allowed to fail without taking the ticket down with it. The offline-sync work was the most satisfying because the mobile side was adamant the payload was identical online and offline, and on paper it was. Rather than argue, I minted a real token, reproduced the exact failing request against the production database inside a rolled-back transaction, and ran a matrix of payload variants until the timezone case lit up. Seeing the same body pass naive and 500 on `+05:00` was the whole answer — the difference wasn't the body the developer typed, it was how the offline queue re-serialized it. If I did it again I would design for serverless and offline-first from day one, rather than retrofitting both after they broke. I would also add a small contract test that fires timezone-aware and naive timestamps at every create endpoint, because that single class of bug cost more debugging time than anything else in the project.

### Outcome

Shipped and running in production on Vercel — the live service reports version 2.2.1 and a healthy database, Firebase, and storage on its root health check. The backend covers 56 endpoints across 11 routers over a 9-table Postgres schema, with a 7-step Alembic migration chain and 50 unit/integration tests run in CI across the Dev, Staging, and main branches. Core capabilities delivered: JWT auth with account lockout after 5 failed attempts and force-expire session invalidation, six-role RBAC, the full ticket lifecycle including a `pending → acknowledged → in_progress → completed → resolved` flow with self-claim, camera-based proof-of-completion uploads to S3/R2, fuzzy duplicate detection, offline-sync idempotency, Firebase push plus email notifications, an append-only audit trail, and scheduled deadline alerts, monthly reports, and archival via Vercel Cron. - Users onboarded / active: the old WhatsApp/paper process:

### Overview

Square One Task Management replaces an ad-hoc (WhatsApp/verbal) workflow with a structured digital system covering task creation, multi-level assignment, execution, camera-proof completion, resolution, analytics, and full audit logging across a 24/7 operational mall. The backend is FastAPI (Python 3.11+) with SQLModel ORM and 9 PostgreSQL tables. Six role tiers (Super Admin singleton enforced by partial index → Admin → HOD → Dept Manager → Supervisor → Employee) govern every endpoint. Class 1 tickets flow from any department to another; Class 2 are internal. Proof of completion requires a live camera capture — gallery upload is blocked at the API level. Background jobs handle monthly performance email reports and deadline alerts. The system supports 102 seeded real employees across 9 departments.

### Features

- 6-tier RBAC (Super Admin singleton via partial index → Admin → HOD → Dept Manager → Supervisor → Employee)
- Two ticket classes: cross-department (Class 1) and internal (Class 2)
- Four-status lifecycle: Pending → In Progress → Completed → Resolved
- Subticket system with circular dependency prevention; parent cannot resolve until all children resolve
- Camera-only proof of completion — gallery upload blocked at API level; supports photo (≤5 MB) and video (≤60 s)
- 16 push notification event types across FCM (Android), APNs (iOS), in-app, and email channels
- Append-only audit log with separate DB role (INSERT + SELECT only) — tamper-proof by design
- Analytics: per-ticket KPIs, department-wise resolution times, employee performance, CSV/PDF export
- Automated monthly performance email reports and 48h/72h deadline alerts via APScheduler

### Stack

FastAPI, Python 3.11+, SQLModel, SQLAlchemy (async), PostgreSQL (Neon), Alembic, JWT, Firebase FCM, APNs, Cloudflare R2, APScheduler, Flutter, Vercel

## Lexnis Residential

URL: https://muhammadsufiyanbaig.vercel.app/projects/client/lexnis-residential
Category: client (featured)
Built for: Innovative Widget
Timeline: Apr 2026 – Jul 2026 (ongoing)
Role: I own the backend solo: the FastAPI + SQLModel service in `Server/`, 17 SQLModel tables, 15 Alembic migrations, ~165 REST + WebSocket endpoints across auth, admin, customer, officer, chat and notification routers, plus the real-time layer (WebSocket connection manager) and the Firebase push integration. I designed the schema, the SOS lifecycle state machine, and the concurrency-safe reference-number scheme. I did not build the Flutter mobile app or the Next.js admin panel — those are separate repos this backend serves.
Links: Live: https://play.google.com/store/apps/details?id=com.lexnis.app

Production backend for a residential security SaaS — async FastAPI + WebSocket server with real-time SOS emergency system, shift management, area group chat, AWS S3 CSV exports, and fully automated Docker CI/CD pipeline on AWS EC2 with health-check rollback.

### Problem

A residential estate needs a way for residents to raise an emergency and have an on-duty guard actually respond, with a record of what happened afterwards. Before this, the "system" was a phone call or a WhatsApp group — no way to know which guard was assigned, whether they were on their way, when they arrived, or what the incident actually was. Nothing was logged, so there was no accountability trail when a resident asked "what happened to my alert last Tuesday?" Guards, residents and the estate admin were all working off memory and screenshots.

### Constraints

Three clients consume one backend — a customer app, an officer app (both Flutter), and an admin panel (Next.js) — so the API had to be the single source of truth and every state change had to reach all three in real time. The target host is serverless (NeonDB Postgres over `asyncpg`), which rules out anything that assumes a long-lived process owns the database. Guards and residents are on phones with flaky connectivity, so "real-time" could not mean "only works while the socket is open." And this is an emergency product — a dropped or ambiguous SOS is a safety failure, not a bug ticket.

### Architecture

The service is a thin-route / fat-controller FastAPI app: routers do auth,
validation and HTTP status; all business logic lives in controllers
(`officer_controller`, `customer_controller`, `admin_controller`, …). Everything
is async end to end against Postgres via `asyncpg`, because the same request that
writes an SOS also fans it out to live sockets and fires a push — blocking I/O
anywhere in that path stalls the event loop.

The two decisions that shaped everything: **one unified `messages` table** and a
**dual real-time delivery path**. Rather than separate tables for group chat,
resident↔admin queries, SOS channels and internal admin chat, there is a single
`messages` table keyed by `channel_type` + `channel_id`. That means one query
path, one media pipeline, one pagination implementation for every kind of
conversation — at the cost of the table meaning nothing without its discriminator,
which I accepted. For delivery, live updates go over an in-process WebSocket
`ConnectionManager` (an `asyncio.Lock`-guarded dict of channel → sockets), and the
*same* events also go out as Firebase Cloud Messaging pushes. WebSocket handles
"resident is staring at the screen"; FCM handles "resident's phone is in their
pocket." The push path is deliberately fire-and-forget — it logs and swallows
every failure and never raises, because a Firebase outage must not block an SOS
from being written.

Media (chat images, incident attachments) is stored in S3 by key, not by URL:
the DB holds `media_key` and a presigned URL is generated fresh on every read with
a 1-hour expiry, so no long-lived public links leak.

```flow
Client :: Flutter customer app POSTs /customer/sos; officer & admin apps hold WebSockets open
API layer :: FastAPI /api/v1 — JWT auth dependency, role guard, slowapi rate limit, request validation
Controller :: SOS state machine (SENT→RESPONDED→PENDING_APPROVAL→RESOLVED) + incident-report creation as side effects
ConnectionManager :: In-memory channel fan-out — broadcasts to sos:{area_id}, sos:admin, notifications:{customer_id}
FCM :: firebase-admin push to the same recipients for backgrounded/offline devices, fire-and-forget
Data store :: NeonDB Postgres via asyncpg — sos_alerts row, sos_incident_reports, atomic reference_counters
S3 :: incident/chat media by key; presigned GET URL minted per read
```

### Challenges

- The SOS lifecycle is a state machine with a human in the middle: An alert moves SENT → RESPONDED (officer acknowledges) → officer sets ETA → marks arrived → resolves. Resolving does *not* close it: it writes an incident report and moves the alert to PENDING_APPROVAL, then the *resident* approves or rejects the resolution, with a consent/rejection branch when they disagree. Each transition guards its predecessor (you cannot mark arrived before an ETA is set, cannot resolve before arriving) and each fans out its own WebSocket event and push. The hard part was that "resolved" is the officer's claim, not the truth — the resident has the final say — so the model carries separate `resolved_at`, `customer_approved_at`, `rejected_at`, `rejection_reason` and `resolution_consent` columns instead of a single boolean.
- Postgres native enums fought SQLModel binding: SQLModel created the DB enum types from the Python enum *member names* (uppercase, `SOS_CSV`), but str-enum query binding sends the *values* (lowercase, `sos_csv`) — so every filter on those columns silently mismatched. Adding a new status was worse: `ALTER TYPE … ADD VALUE` cannot run inside a transaction, and Postgres cannot remove an enum value, so the migration (004) is a one-way `ADD VALUE IF NOT EXISTS` with a no-op downgrade. I eventually gave up on native enums for the export type and migrated the column to plain `VARCHAR` with a `USING LOWER(...)` cast (015). Lesson learned the expensive way: with SQLModel + Postgres, string columns beat native enums.
- Reference numbers had to be unique under concurrency: Every SOS, incident and export needs a human-readable `PREFIX-YYYYMMDD-NNNN` id that two simultaneous requests can't collide on. A `SELECT max()+1` would race. I used a `reference_counters(prefix, counter_date)` table and a single atomic statement — `INSERT … ON CONFLICT (prefix, counter_date) DO UPDATE SET last_seq = last_seq + 1 RETURNING last_seq` — so the database does the increment and hands back the sequence in one round trip, with no application-level lock.
- Real-time presence and history over one socket: The chat WebSocket has to authenticate a JWT passed as a query param, verify the connecting user actually belongs to that area group (or has an active shift, for officers), send the last 50 messages as history on connect, then serve cursor-based `load_history` pages on demand — all while broadcasting new messages, an area-list metadata update, and an FCM topic push per message. Getting the media URLs right meant regenerating a presigned URL for every historical message on the way out, since stored keys outlive any single signed link.
- Startup can't assume the database is reachable: On a serverless target the app may boot before the DB is warm. The lifespan handler seeds the admin account and app-version row inside a try/except that logs and continues, flipping an `app.state.database_ready` flag that `/health` reports as `ok` vs `degraded`, rather than crash-looping the container on a cold database.

### New ground covered

- Firebase Cloud Messaging (firebase-admin): First time wiring server-side push. I learned it is not one API but several — multicast to device tokens, topic broadcast, single-token — each needing explicit Android priority and full APNS payload config or iOS silently drops the notification when backgrounded. I also learned to treat invalid/unregistered token codes as a signal to prune dead tokens rather than retry them.
- WebSockets as a first-class channel, not an afterthought: I had used request/response APIs; running a channel manager with per-channel subscriber lists, join-multiple-channels-per-socket, typing/read events and dead-socket cleanup under an async lock was new. I understand now why presence and history are the hard parts, not the message plumbing.
- Alembic enum surgery: Before this I treated migrations as add-a-column. This project taught me the operations Postgres won't do transactionally or reversibly, and how to write a defensible one-way migration with a documented no-op downgrade.
- Async S3 with boto3: boto3 is synchronous, so every call is wrapped in `asyncio.to_thread` to keep it off the event loop, and the client is built lazily and cached — with a 503 if storage credentials aren't configured rather than a crash at import.

### Takeaways

- Modeling "resolved" as an officer claim that a resident must confirm, rather than a single status flag, is what turned the SOS flow from a feature into an accountability record.
- Native Postgres enums are a liability with SQLModel; a `VARCHAR` plus an application-level enum avoids a whole class of name-vs-value binding bugs.
- Real-time delivery needs two paths — a socket for the foreground and a push for the background — because "the connection is open" is never a safe assumption on mobile.
- Push notifications fail constantly (dead tokens, provider hiccups) and must be fire-and-forget; letting FCM raise into the request path would have made a Firebase outage look like an SOS outage.
- Concurrency-safe sequencing belongs in the database (`ON CONFLICT … RETURNING`), not in Python where two requests will eventually race.

### Journey

It started as a straightforward CRUD backend — users, areas, shifts, a chat table — and the SOS was going to be "insert a row, notify a guard." That lasted until the first real conversation about accountability, when it became obvious that an alert without a lifecycle is just a louder phone call. The migration history is honest about this: the very first migration adds ETA/arrived/resolved timing columns to a table that already existed, because I hadn't seen the state machine coming. What I got wrong early was trusting the ORM's defaults. I let SQLModel generate native Postgres enums and moved on, and it worked in every test until a filtered query returned nothing in a real database and I couldn't see why. The DB had `SOS_CSV`, the query sent `sos_csv`, and nothing errored — it just quietly matched zero rows. Tracking that down, and then discovering Postgres won't let you remove an enum value or add one inside a transaction, was the week that taught me to stop reaching for native enums. The moment it clicked was the customer-approval step. Once I stopped treating "resolved" as a boolean the officer flips and modeled it as a claim the resident confirms — with its own rejection and consent branches — the whole thing made sense. The rating system, the incident report, the reference numbers all hung off that spine cleanly, because there was finally a real object with a real end state to attach them to. If I did it again I'd design the SOS state machine and the reference-number scheme on day one instead of retrofitting them across a dozen migrations, and I'd use `VARCHAR` for status columns from the start. I'd also decide the WebSocket-plus-FCM dual delivery contract up front rather than discovering per-feature that a socket broadcast alone never reaches a backgrounded phone. The one thing I'd keep exactly as-is is the fire-and-forget push layer. Every time Firebase misbehaved in development, the SOS still wrote, still broadcast, still health-checked green — and that separation is the difference between a degraded feature and a down product.

### Outcome

Shipped as the live backend for the Lexins platform, serving a Flutter customer app, a Flutter officer app and a Next.js admin panel from one API. What's in the repo: 17 tables, 15 Alembic migrations, ~165 REST and WebSocket endpoints, a full SOS lifecycle from raise through officer response to resident-confirmed resolution with an attached incident report, unified real-time chat, and dual WebSocket + FCM delivery. It is backed by 730 test functions across 42 test modules covering auth, admin analytics, the SOS flow, ratings, notifications and media handling. As of July 2026 the client-feedback round (daily occurrence logs, officer-initiated SOS, map tracking, multi-officer dispatch) is planned and in progress.

### Overview

Lexins SOS is a comprehensive community management system for residential estates. The backend is a fully async FastAPI (Python 3.13) + WebSocket server with 60+ REST endpoints and 3 WebSocket channels, serving Admin, Customer, and Security Officer roles via JWT auth with refresh token rotation. The SOS emergency system broadcasts real-time alerts to active officers via WebSocket and tracks the full lifecycle: SENT → RESPONDED → PENDING_APPROVAL → RESOLVED with ETA, arrival time, resolution time, and a post-SOS officer rating. AWS S3 CSV exports use boto3 presigned URLs with a graceful BYTEA fallback. The CI/CD pipeline runs: ruff lint → pytest (200+ tests, Postgres 16 container) → docker buildx → push to AWS ECR → SSH to EC2 → migrate → health-check rollback.

### Features

- Real-time SOS emergency system — WebSocket broadcasting, full lifecycle (SENT → RESPONDED → PENDING_APPROVAL → RESOLVED)
- 3 WebSocket channels: area group chat, SOS alert channel, admin DM queries
- Shift scheduling with auto-add/remove of officers to area group chats on shift lifecycle events
- Service request lifecycle (PENDING → IN_PROGRESS → COMPLETED / CANCELLED) with soft deletes
- AWS S3 CSV exports via boto3 presigned URLs (1-hour expiry) with BYTEA fallback
- SOS analytics — trend charts, area-wise breakdown, response-time distribution, KPI cards
- Refresh token rotation — every refresh issues a new token and revokes the old one
- Admin seeding on startup from env vars — no manual DB insertion needed
- 200+ async integration tests (pytest + httpx) with Postgres 16 service container in CI

### Stack

FastAPI, Python 3.13, SQLModel, SQLAlchemy (async), asyncpg, PostgreSQL (Neon), WebSocket, JWT, bcrypt, boto3, AWS S3, AWS EC2, AWS ECR, AWS IAM, Docker, GitHub Actions, Firebase FCM, pytest, Uvicorn

## Alpha Trace

URL: https://muhammadsufiyanbaig.vercel.app/projects/personal/alpha-trace
Category: personal (featured)
Timeline: Jul 2026 (11-day intensive build on a prior academic data-mining base)
Role: I owned the system architecture, the ML pipeline, and the full-stack dashboard: registry-driven onboarding (`markets.json` + `onboard_market.py`), the walk-forward validation protocol and per-market gating in `xgboost_model.py`/`company_model.py`, the live-signal ledger and weekly regression-guarded retrain (`predict_live.py`, `refresh_market.py`), the GraphQL API (40 resolvers, 52 types in `dashboard/api/`), and all 19 screens of the Next.js terminal including the canvas-based orthographic globe. A second team member (backend/data engineering) contributed to the data-ingestion layer. Every architectural decision, the validation discipline, and the eventual removal of two markets after they failed on evidence were mine to make and defend.
Links: Code: https://github.com/muhammadsufiyanbaig/Alpha-Trace-2.0

Full-stack data mining research platform analyzing how geopolitical events and community sentiment causally influence stock market behavior across 5 global exchanges (2.2M+ hourly records) using Granger causality, XGBoost, Neo4j graph analysis, and a GraphQL dashboard.

### Problem

Retail and institutional "AI predicts the market" products backtest well and then fail live, and nobody ever discloses when the underlying signal stops working — so risk and compliance teams can't adopt them. Separately, there is no accessible tool that connects a specific news event or public figure's statement to a measurable, evidenced market move; that connection lives in analysts' heads as folklore, redone by hand in spreadsheets every time it's needed.

### Constraints

Free-tier-only data: GDELT and Yahoo Finance rate-limit aggressively (GDELT returns HTTP 429 within minutes of sustained use), so every scraper had to be resume-safe and tolerate multi-day partial completion. The project ran as a solo-architected build against a hard hackathon/demo deadline, on a single Windows development machine, with no dedicated infra budget — which ruled out a database server and forced a file-based (CSV/parquet) architecture that still had to serve a live dashboard without restarts.

### Architecture

The system is split into three layers that don't share process boundaries on purpose: a pipeline layer that writes plain files (CSV/parquet/JSON), a GraphQL layer that reads those files fresh on every request, and a Next.js terminal that queries GraphQL. The reason for the file-based store instead of a database is explicit in the code: at this scale (~1.65M rows) a database server is operational overhead with no query the files can't already answer in milliseconds, and file mtimes give "the API always serves what's on disk" for free — no cache-invalidation logic, no restart-on-deploy step. The tradeoff is real: there's no concurrent-write safety, which is why the live ledger (`live_signals.parquet`) is strictly append-only and every write path (`predict_live.py`, `refresh_market.py`) treats existing rows as immutable once an outcome resolves.

The second architectural decision that shapes everything downstream is the market registry (`markets.json`): every pipeline script, the API, and every dashboard dropdown reads which markets exist from that one file instead of hardcoding a list. That decision was made after discovering — repeatedly, across several commits — that a hardcoded market list in one file and not another caused dashboards to silently disagree about how many markets existed. `onboard_market.py` turns adding a market into one CLI command that runs the whole chain (scrape → features → train → validate → verdict) against that registry.

```flow
Scheduler :: Windows Task Scheduler fires update_data.py nightly (03:45) and refresh_market.py weekly
update_data.py :: Appends new price/FX/VIX rows and re-fetches the current GDELT month; resume-safe, tolerates partial 429 failures
feature_engineering.py :: Rebuilds tone/return/volatility features per market from markets.json's registry, no hardcoded market list
predict_live.py :: Trains LR+XGBoost on all history with the SAME validated per-market gates as the backtest engine, emits one forward signal per market
live_signals.parquet :: Append-only ledger — signal_ts is written BEFORE the outcome is knowable; rows are never edited, only outcome-backfilled later
GraphQL API (Strawberry) :: loaders.py reads the parquet fresh (mtime-checked), computes accuracy through one canonical stats function shared by every screen
Next.js dashboard :: /signals and /performance query the ledger; a signal below its market's validated confidence gate is shown as an abstention, not a guess
```

### Challenges

- Pages disagreeing about their own numbers: Different screens computed "high-confidence accuracy" with slightly different windowing logic, so the same market showed two different accuracy figures depending on which page you were on. I traced it to duplicated statistics code and replaced every accuracy computation across the API with one function (`_hc_stats`/`_val_slice` in `loaders.py`) so no two screens can disagree by construction.
- A retrain that quietly got worse: The first live run of the weekly retrain script dropped one market's validated accuracy by more than 5 percentage points against the recorded baseline. Rather than trusting the new numbers, `refresh_market.py` snapshots the model outputs before retraining and calls `verify_model.py` as a hard gate — if any market regresses past a threshold, it restores the snapshot automatically and the bad model never reaches the dashboard. This fired for real on the first run, not in a test.
- A UK signal that looked promising eight times in a row: Feature engineering, gate tuning, calibration, sector sub-models, an alternate horizon, mid-cap tickers, and hourly intraday bars were each tried against UK data and each looked like an improvement on the training folds — and each one collapsed when checked against the frozen validation folds that were never touched during tuning. The discipline that mattered was refusing to let a promising dev-fold number override a bad validation-fold number, even on the eighth attempt. The market was eventually dropped rather than shipped with an unsupported claim.
- Free-tier rate limiting breaking multi-day scrapes: GDELT throttles per IP within minutes of sustained requests, which made a naive scraper unusable for pulling years of history across five countries. Scrapers were rewritten to skip already-fetched months and resume cleanly after a 429, so a scrape could be stopped and restarted (including across IP changes) without re-downloading or duplicating data.
- A shell-escaping bug that silently zeroed out a whole feature: A regex word-boundary pattern (`\b`) built through a heredoc got corrupted into a literal backspace character by the shell, which made a headline-matching filter match nothing — the globe's news-evidence panel returned zero headlines for every market with no error thrown anywhere. It only surfaced because the empty result looked suspicious enough to check by hand; there was no test that would have caught it.

### New ground covered

- Walk-forward validation with a frozen holdout, not just a train/test split: I'd used cross-validation before but not the specific discipline of splitting folds into a "dev" region you're allowed to tune against and a later "validation" region you check exactly once and never re-tune on. What I understand now that I didn't before: an accuracy number is not trustworthy on its own — it has to state which fold produced it and whether that fold was ever touched during tuning, or the number is meaningless.
- Wilson score confidence intervals for small-sample classification accuracy: Reporting "80% accuracy" without its interval is close to reporting nothing when the sample is 20 predictions. I now default to citing the CI-low bound for any accuracy claim under a few hundred samples, not the point estimate.
- Brown-Warner abnormal-return event studies: Building `figure_impact.py` was the first time I'd implemented an event-study methodology (mean-adjusted abnormal returns against an estimation window, per-event t-statistics, a confounding-event flag) rather than just correlating a feature against a return series.
- GraphQL as the sole boundary between a file-based pipeline and a live frontend: Using Strawberry/FastAPI to make a directory of parquet files behave like a live API — including making the API discover new markets from disk without a restart — was new; the pattern of mtime-based freshness instead of a cache-invalidation scheme is something I'd reach for again.

### Takeaways

- An accuracy number without its sample size and confidence interval is not a claim, it's a guess wearing a decimal point.
- A model that abstains honestly is more valuable to a real user than one that never admits uncertainty — the UK market's rejection became the strongest credibility story in the whole project, not a weakness to hide.
- Free public data sources (GDELT, Yahoo, Guardian) can replace a paid data budget entirely, but only if every scraper is written to survive rate-limiting from the start, not patched in afterward.
- A regression guard that can auto-rollback a bad model is worth building before you need it — the first time it fired was in production, not in a drill.
- Duplicating a small calculation (like an accuracy formula) into more than one file is how a dashboard ends up silently lying to itself; one canonical function per computed number is worth the refactor.

### Journey

This started as a five-market academic sentiment-mining exercise and turned into something closer to a real trading terminal partway through, once it became clear that "does news sentiment correlate with returns" wasn't an interesting enough claim on its own — the interesting question was whether the model could be trusted to know when it didn't know. That reframing is what pulled in the validation protocol, the confidence intervals, and eventually the honest abstention framework that ended up being the actual product. The UK market cost the most time and taught the most. Every single lever I could find — new features, recalibrated gates, sector-level sub-models, a longer prediction horizon, mid-cap tickers instead of large-cap, hourly bars instead of daily — looked like a fix on the training folds and then fell apart on the validation folds, one after another, across eight separate attempts. It would have been easy to ship the sixth or seventh attempt's dev-fold number and call it done; the fact that none of them held up on data the model hadn't seen is the reason I trust the markets that did pass. The moment it clicked was building the weekly regression guard and watching it actually fire on its first real run — a retrain came back objectively worse on three markets, and the system caught it and rolled itself back before anything bad reached the dashboard, with no one watching. That's when the project stopped feeling like a model and started feeling like a system with a safety net. If I did it again, I'd write the canonical-stats module and the shared market registry on day one instead of after discovering the disagreement bugs they were built to fix — both were retrofits onto code that had already grown five or six independent copies of logic that should have been one function from the start. I'd also add automated tests around the statistics functions and the ledger's append-only guarantee; right now that correctness is enforced by convention and by me checking the numbers by hand, which doesn't scale and didn't catch the shell-escaping bug until it was visible in the UI.

### Outcome

A working system covering 5 markets (NYSE, TSX, TSE, ASX, KOSPI) after two (UK, Hong Kong) were deliberately removed following documented, repeated validation failures. The production model store holds roughly 1.65M walk-forward predictions across 468 companies; the market-level engine's frozen out-of-sample validation reaches 79.7% accuracy on its strongest market (n=59) and a pooled 62.3% across all five markets (n=738) against a 50% baseline. The experiment log (`Community/Output/Model/experiments.md`) records 23 dated attempts, the majority of them rejections. The live forward-signal ledger and the weekly guarded retrain were both running unattended on a schedule as of this writing, with the regression guard having already triggered one real rollback.

### Overview

AlphaTrace investigates whether community-level signals — GDELT news sentiment, World Bank macroeconomic indicators, OECD data, Reddit discourse — causally influence stock returns and volatility across NYSE, LSE, TSE, TSX, and HKEX over 5 years (2020–2025). The pipeline: async multi-source scrapers with rate-limiting and checkpoint-resume → 7-module Neo4j ETL (2,500+ nodes: Company, Country, Event, CommunityIndicator) → 80+ engineered features per country → 8 data mining algorithms → FastAPI + Strawberry GraphQL API (20+ query types) → Next.js 14 dashboard (9 pages, Nivo / ECharts / Recharts / Tableau). Algorithms: Granger Causality, VAR Impulse Response, Event Study (CAR), K-Means (stocks + market regimes), XGBoost (binary next-day direction, TimeSeriesSplit validation), SHAP explainability, PCA (52 macro indicators → 5 components).

### Features

- Multi-source async scrapers — GDELT DOC 2.0, World Bank API, OECD SDMX, Reddit PRAW, Guardian/NYT/GNews with checkpoint-resume
- 7-module Neo4j ETL — 2,500+ nodes with INFLUENCED_BY / CORRELATED_WITH edges
- 80+ engineered features per country: daily returns, volatility regimes, GDELT tone lags, macro PCA components
- Granger Causality + VAR Impulse Response — does community data cause stock returns?
- Event Study (CAR) — cumulative abnormal returns in [-5, +10] windows around GDELT tone spikes
- K-Means clustering (k=4 stocks, k=3 market regimes: Growth / Stagnation / Crisis)
- XGBoost binary classifier with TimeSeriesSplit cross-validation + SHAP explainability
- FastAPI + Strawberry GraphQL API — 20+ typed query endpoints exposing all algorithm outputs
- 9-page Next.js 14 dashboard — Granger heatmaps, VAR impulse graphs, regime timelines, SHAP importance

### Stack

Python 3.10+, FastAPI, Strawberry GraphQL, Neo4j 5.0+, MS SQL Server, TypeScript, Next.js 14, React 18, Apollo Client, Tailwind CSS, Zustand, pandas, NumPy, scikit-learn, XGBoost, SHAP, statsmodels, yfinance, PRAW, Nivo, ECharts, Recharts, Tableau Embedding API

## Clutch Darts

URL: https://muhammadsufiyanbaig.vercel.app/projects/client/clutch-darts
Category: client (featured)
Built for: Element 2.0158 UG & Co KG
Timeline: Jul 2025 – Jul 2026 (ongoing)
Role: I own the backend solo. That covers 65 route handlers across 17 API groups, the 26-table Postgres schema and its 34 migrations, and every query behind them: email/Google/Apple auth, match ingest for four game modes, versioned training plans, streaks, the statistics module, and push notification scheduling. 351 commits on the current branch are mine. The iOS app is built by someone else, so every request and response shape in here is a contract I agreed with a client developer and then had to keep working while the data model changed underneath it.
Links: Live: https://www.clutchdarts.com/

Backend for an international dart training and game tracking application based in Germany — handling real-time game sessions, personalized training plans, and player statistics.

### Problem

A darts player practising alone has no way to tell whether they are actually improving. A three-dart average written in a notebook does not answer whether your checkout percentage is better than last month, or whether you only ever win against the bot. Players were coping with paper and spreadsheets, which nobody sustains long enough to see a trend. On top of that, "I'll practise three times a week" fails silently — there was nothing tracking whether you actually showed up.

### Constraints

The app is offline-first: a player throws in a garage with no signal, and sessions sync later, sometimes twice, sometimes out of order — so idempotency is a requirement on every write endpoint, not a nicety. Everything a player sees is in their own timezone, so day boundaries cannot be computed in UTC and cannot be trusted from the client either. Training reminders need minute-level accuracy, which on Vercel means a cron that runs every minute and finishes inside its budget. And the hardest one: training plans are editable, but streaks are historical — changing a plan today must not rewrite whether you kept your streak last Tuesday.

### Architecture

This is a Next.js App Router API on Vercel with Neon Postgres over Drizzle,
deployed through a debug → dev → staging → main branch flow with a separate
database per environment. It is deliberately one deployment rather than a service
split: the difficulty here is data modelling and query correctness, not
throughput, and splitting would have added network boundaries between things that
need to be consistent with each other. Routes are kept thin — authenticate,
validate with zod, shape errors — and the work lives in `lib/`, grouped by domain
(`statistics`, `streaks`, `notifications`, `auth`, `recommendations`). Rate
limiting sits in `lib/utils/rate-limit.ts` on Upstash Redis with an in-memory
fallback for local dev, because Vercel functions do not share process memory and
a per-instance counter would not have limited anything.

Two decisions shape most of the code. First, aggregation stays in SQL. A year
view is thousands of throws, and bucketing plus averaging them in Postgres — with
`AT TIME ZONE` applied inside the query so the day boundaries are the player's —
keeps the endpoint fast on a cold function; the controller only fills gaps, so a
week with no darts still returns seven labelled points with `null` values and the
client never reasons about missing days. The cost is long SQL that was duplicated
per period and per game mode, which I paid down by extracting shared fragments
into constants rather than by moving the work into TypeScript. Second, training
plan items are versioned rather than mutated: `valid_from` and `deactivated_at`
on `training_plan_items`, `deactivated_at` on `training_plans`. Editing a plan
retires the old row and writes a new one, so "was this a planned training day?"
stays answerable for any past date. Hard updates would have been far simpler and
would have silently corrupted every historical streak.

```flow
iOS client :: Queues completed matches offline, syncs in batches when a connection returns
POST /api/game/around-the-clock/sessions :: JWT auth, zod validation, dedupe on clientGameId before any insert
game_sessions + darts_throws :: The match summary and every individual dart, Postgres on Neon
training_sessions :: The same match when played from a plan — separate legacy shape, different JSON keys
GET /api/statistics/summary :: Validates period and IANA timezone, then delegates immediately
lib/statistics/stats-queries.ts :: One pass per game mode, UNIONs both session tables, buckets in the player's timezone
lib/statistics/stats-controller.ts :: Fills empty buckets, weights averages, builds per-variant summaries and charts
```

### Challenges

- Streaks against an editable plan: This is the most-rewritten code in the repo — roughly nineteen fix commits and one outright revert. Each failure was specific: streak days counted per session instead of per local day, active plans truncated by a sibling plan's `created_at`, the year summary leaving planned days out of the denominator, the plan's own creation day excluded from missed-training math. What finally worked was adding an explicit validity window (`deactivated_at`) to plans and treating "was today a training day" as a pure function of plan state as-of that date, rather than as of now.
- Plan edits that quietly deleted history: Four consecutive commits fight over the same semantics — whether creating a plan replaces existing ones. It started as replace-all, which wiped user plans whenever the app generated a recommended one. It became additive, then additive-but-retire-only-the `app_recommended` plan, then additive with day-scoping preserved on update. The root cause was that one endpoint was serving two different intents, app-generated and user-authored, distinguished only by a `plan_source` column added later.
- One match, two tables: The same game lands in `game_sessions` if played freely and in `training_sessions` if played from a plan, with different JSON shapes for identical numbers. Every stats query has to `UNION ALL` both and normalise as it reads. I considered migrating to one table and rejected it: the write paths are shared with a shipped iOS client I do not control, so the normalisation had to live in the read layer instead.
- Opponent classification was silently wrong: The training-session branch treated "an opponent object exists" as proof the opponent was the Kaspa bot — but those routes write that object for solo games too. So every human match and every solo run counted as a bot match; win-rate-vs-human returned `null` and the bot win rate was computed over solo games. I found it by classifying every production row under both the old and new rule and diffing, which also proved the fix changed nothing for genuine bot matches.
- No test suite: There are no automated tests in this repo. For a correctness-heavy read API that is a real gap, and it is why verification here means writing throwaway harnesses that post every game mode × variant × opponent type through the live endpoints, assert the response, then delete what they created. It works, but it is a slow substitute for tests and it does not protect the next change.

### New ground covered

- Bitemporal-ish plan versioning: Keeping `valid_from` / `deactivated_at` so a mutable plan can still answer questions about the past.
- Timezone-correct SQL aggregation: Bucketing on `(completion_time AT TIME ZONE $tz)::date` rather than trusting UTC or the client's own dates.
- Distributed rate limiting on serverless: Upstash Redis with an in-memory local fallback, because per-instance counters do not limit anything across Vercel functions.

### Takeaways

- Anything a user can edit needs a validity window from day one if anything else derives history from it — retrofitting `deactivated_at` onto training plans cost more than designing it in would have.
- A summary endpoint returning `null` looks identical whether the metric is genuinely empty or the rows were misclassified, so every "no data" path needs to be provable from the raw rows before you believe it.
- Reading the write path before changing a read query would have saved me a wrong fix — I had already reordered a classification `CASE` before noticing the column it read was itself resolved by the route that writes it.
- One endpoint serving two intents will grow a boolean, then a column, then four fix commits; splitting app-generated and user-authored plan creation earlier would have avoided all of them.
- Diffing old versus new logic across every production row is a faster correctness argument than any unit test I could write for SQL this shaped.

### Journey

I started this expecting a CRUD backend — record matches, return them — and assumed statistics would be the easy part. Statistics and streaks between them account for most of the rework in the repo. The first real mistake was writing the stats SQL inline in route files. It worked, so I did it again in a second route, and the two copies drifted almost immediately: one read the three-dart average from one JSON path and the other from a different one, so the dashboard and the statistics screen disagreed for a while before anyone noticed. Pulling it into `lib/statistics` and splitting queries from calculations from pure helpers was the change that made the module maintainable, and I should have done it at the second copy rather than the tenth. Streaks were worse, and the git history does not hide it — there is a commit literally called `streak revert`, another called `fix all identified 14 issues`, and a run of narrowly-scoped fixes after each of those. Every time I fixed the streak calculation, the next plan-editing feature broke it again. The moment it clicked was realising I was not fixing a streak bug at all: I was repeatedly discovering that a mutable training plan cannot answer questions about the past. Once plans and plan items carried their own validity windows, most of that class of bug stopped being possible. The most recent one taught me a different lesson. A user reported that win rate versus human never appeared for one game mode, and my instinct was that the aggregation was wrong. It was not — the classification feeding it was, and it had been wrong for every game mode. It only got reported for one because that mode's human matches happened to arrive through the other table. What I would do differently: normalise the two session tables at the write layer on day one, even at the cost of an awkward early migration. Every read query in this repo pays a tax for that split, and the tax grows with each new game mode.

### Outcome

Shipped and running in production across three environments (dev, staging, production), each with its own Neon database, deployed from a debug → dev → staging → main branch flow. Provable from the repo: 65 API route handlers across 17 groups; 26 tables and 34 migrations; four game modes recorded, three of them surfaced in statistics (X01, Around the Clock, Bob's 27) across nine variants; rate limiting at 1000 requests per hour per client with a 5-per-hour limit on OTP; a notifications cron running every minute with a 300-second function budget; a GitHub Actions job backing up all three databases to Vercel Blob every Sunday at 02:00 UTC; 351 commits between July 2025 and July 2026. Ten `TODO` markers remain, all of them the same unfinished thread: subscription status is hardcoded at the auth endpoints pending the RevenueCat integration.

### Overview

Clutch Dart is an international dart training and game tracking application based in Germany. The backend powers a full-featured mobile app experience — handling real-time game sessions, personalized training plans, player statistics, and a subscription-based ecosystem. Built with Next.js App Router and TypeScript, using PostgreSQL on Neon with Drizzle ORM. The system includes JWT dual-token authentication with Google/Apple OAuth, a full X01 game engine, weekly training plans with timezone-aware scheduling, and RevenueCat-integrated subscriptions.

### Features

- Authentication System — JWT dual-token auth, Google/Apple OAuth, OTP-based passwordless login
- Game Engine (X01) — Full game session lifecycle with throw recording and live state tracking
- Training Module — Permanent weekly training plans with daily status tracking
- Recommendations Engine — Score-based X01 checkout recommendations
- Dashboard & Analytics — Aggregated player stats and historical chart data
- Referral & Subscription System — RevenueCat-integrated subscriptions with referral extensions
- Content System — Articles with ratings, comments, and social sharing
- Offline-First Support — Client-side UUID tracking for reliable game sync
- 31 REST API endpoints organized by domain across 30 database tables

### Stack

Next.js, TypeScript, PostgreSQL (Neon), Drizzle ORM, JWT, OAuth 2.0, Zod, Nodemailer, RevenueCat, Firebase FCM

## Recrot

URL: https://muhammadsufiyanbaig.vercel.app/projects/client/recrot
Category: client (featured)
Built for: Innovative Widget
Timeline: Oct 2025 – Jul 2026 (10 months, ongoing)
Role: Two-person development team. I was one of the two developers and the one who reviewed and merged the feature branches into `dev`/`main` (26 of the 297 commits are my merge commits). The subsystems I built end to end: email verification and email-based 2FA (`lib/utils/auth/email-2fa.ts`, `app/api/auth/email-2fa`), the whole notification pipeline (in-app documents, the Nodemailer email route, FCM push, and the admin broadcast panel), the tutor payment and subscription-expiry system, tutor search with filtering, job soft-delete/restore, the parent-side hired-tutor and rating screens, and the security-audit remediation pass that added auth guards and closed the silent-failure paths in notifications. I also wrote the platform documentation — SRS, `Docs/api-docs.md`, and the 50-table PostgreSQL target schema (`Docs/schema.md`) for the current migration off Firebase. My co-developer owned the landing pages, most of the employer/admin dashboards, and the job/application UI.
Links: Live: https://recrot.vercel.app/

A full-stack job management platform connecting tutors, parents, and companies with a role-based, subscription-gated system.

### Problem

Tutoring in Pakistan is matched through WhatsApp groups, Facebook posts, and word of mouth. A parent looking for an O-Level physics tutor in their own neighbourhood has no way to see who is qualified, who is nearby, or who anyone else has actually hired and rated. Tutors have the mirror problem: they hear about openings only if they happen to be in the right group at the right time, and they have no verifiable public record of past work. Tutoring agencies sat in the middle taking a cut for doing exactly this manual matching, badly and slowly.

### Constraints

No card payment gateway was available for the business, so every subscription payment had to run through bank transfer plus a screenshot that a human admin verifies — the entire billing state machine had to be built around a manual approval step. The budget assumed Firebase's free and low tiers, which meant per-document-read pricing was a hard design constraint, not an optimisation to do later. The whole platform had to serve four distinct roles (tutor, parent, company, admin) with different dashboards and different permissions, and it had to be ready for a Flutter mobile app to consume the same backend later, which pushed us toward putting the rules somewhere both clients could reach. Two developers, no QA, no test suite.

### Architecture

Next.js 15 App Router with Firebase as the entire backend: Auth for identity, Firestore for data, Cloud Storage for documents and payment screenshots, FCM for push. The consequential decision — and the one I would revisit — was to let client components talk to Firestore directly through the web SDK, with API routes reserved only for things that genuinely need a secret (Nodemailer credentials, the Firebase Admin SDK, admin user creation). That is why there are 76 page routes but only 15 API route handlers. It made the app fast to build and gave us free realtime updates via `onSnapshot`, but it moved the real authorization boundary into `firestore.rules`, which is a language that can express "only the tutor named on this application may update it" and cannot express "only if this tutor has applications left this month."

So enforcement ended up layered, deliberately. `middleware.ts` is a cheap first pass — it checks for the `firebase-auth-token` cookie and redirects, nothing more; the comment in it says as much, full JWT verification happens inside each handler via `getVerifiedToken`. `firestore.rules` is the backstop that a hostile client cannot get past: field-level diffs (`request.resource.data.diff(resource.data).affectedKeys().hasOnly(['status'])` on hire requests), role lookups, and a default deny on everything unmatched. The business rules that rules can't express — subscription validity, application limits — live in shared helpers (`verifySubscriptionServer`, `canPerformAction`) that both the client-side `applyToJob` and the `/api/tutor/apply-job` route call. Rate limiting is a separate concern again: `lib/utils/rateLimit.ts` returns an Upstash Redis sliding-window limiter when the env vars are set and silently falls back to an in-memory `Map` for local dev, because an in-memory counter on Vercel's serverless functions limits nothing.

```flow
Client :: Next.js App Router page (app/tutor/jobs/[slug]) calling Firestore via the web SDK
middleware.ts :: Cookie presence check, role headers, redirect; blocks /api/admin and /api/tutor without a token
applyToJob :: lib/utils/helpers/firebaseHelpers/application.ts — requireRole, duplicate-application check, job-is-active check
verifySubscriptionServer :: Reads subscriptions/{uid}, compares ISO endDate, then canPerformAction against applicationLimit
Firestore :: Writes the application doc, increments jobs.stats.totalApplications and subscriptions.applicationsUsed
firestore.rules :: Independently re-checks the write — tutor role, own tutorId, status must be 'pending'
createNotification :: Writes notifications/{id}, then fans out to email and push with the caller's ID token attached
/api/notifications/send-email + push-notification :: Nodemailer and firebase-admin messaging — the only places server secrets live
```

### Challenges

- Every jobs page load read the entire jobs collection: `getJobs` fetched all documents ordered by `createdAt` and applied category, schooling-system, and status filters in JavaScript. The code comment justified it: doing it server-side would have required composite indexes. Firestore bills per document read, so the cost scaled with collection size times traffic, and the public landing page called `searchTutors` twice per anonymous visitor — a bot hitting `/` cost us money with no login. The fix was to push every filter into `where` clauses, add `limit(20)` with `startAfter` cursor pagination, replace count-by-reading-everything with `getCountFromServer`/`getAggregateFromServer`, and ship the ten composite indexes in `firestore.indexes.json` that this made necessary. Documented projection in `Docs/COST_ESTIMATION.md`: ~4.2M reads/day down to ~200k/day at a 1,000-DAU baseline.
- Plan values were stored with inconsistent casing, which blocked the query we needed: Every job posting notified matching tutors, and finding them meant reading the whole `subscriptions` collection and filtering for paid plans in JavaScript — because `plan` was written as `"Basic"`/`"Pro"` in some paths and lowercase in others, and Firestore's `in` operator is case-sensitive. This is quadratic: subscriptions × jobs posted. I could not simply normalise the field without a data migration and touching every write path, so the interim fix was `where("plan", "in", ["Basic", "Pro"])` matching the actual stored casing, plus `where("status", "==", "active")`. That second clause fixed a real bug as a side effect — expired-but-not-yet-deactivated tutors had been receiving job-match emails they could not act on. The proper normalisation is in the Postgres schema as a strict `subscription_plan` enum.
- Billing with no payment gateway: Tutors upload a bank-transfer screenshot, an admin approves or rejects it, and only then does `verifyPayment` write the subscription with an `applicationLimit` pulled from `paymentSettings`. Upgrades and downgrades have to cancel the old plan before activating the new one, and a tutor can have a pending submission sitting against an already-active plan, so `isPending` had to become a first-class state in `SubscriptionData` separate from `isActive` — several call sites broke when I added it because they were treating "not active" as "free". Expiry runs on three layers: a daily Vercel cron (`0 0 * * *`) that flips expired rows to `inactive`, an `onSnapshot` listener that recomputes on the client, and `verifySubscriptionServer` at the moment of the action, because the first two can both be stale or bypassed.
- The session cookie did not prove anything: The production audit found that `/api/set-session` stored the literal string `"true"` in `firebase-auth-token` rather than the ID token, which meant middleware could confirm *somebody* was logged in and nothing else — and `/api/notifications/[userId]` had no auth at all, so any caller could read any user's notifications by changing the path segment. Fixing it meant introducing `getVerifiedToken`, which verifies the bearer ID token through the Admin SDK, and then adding `decoded.uid!== userId → 403` checks to every handler that takes a user ID. Twenty-three of the twenty-six issues in `Docs/PRODUCTION_AUDIT.md` are now marked resolved; three medium ones (the `apply-job` TODO, four `) are still open and tracked in that file.
- Notification fan-out blocked job creation: A new job notifies every matching paid tutor across three channels — Firestore document, email, and FCM push — and the original loop awaited each tutor in turn, so posting a job with a few hundred matches meant a few hundred sequential round trips while the employer stared at a spinner. `notifyTutorsInChunks` now runs `Promise.allSettled` over chunks of 20. `allSettled` rather than `all` is the point: one tutor with a stale FCM token must not abort the fan-out for everyone behind them, and the same reasoning is why `createNotification` swallows email and push errors rather than failing the notification write.

### New ground covered

- Firestore aggregation queries: `getCountFromServer` and `getAggregateFromServer(sum(...))` were the difference between the admin dashboard reading roughly five thousand documents to display six numbers and reading about twenty index entries. I had been treating Firestore as "documents only" and reaching for a maintained counter field whenever I needed a total; knowing the aggregation API exists changes which denormalisations are worth the write amplification.
- Distributed rate limiting on serverless: My first rate limiter was a `Map` in module scope. It works locally and does approximately nothing on Vercel, where each concurrent invocation gets its own instance — the audit flagged it. Upstash Redis with a sliding window gave shared state, and the fallback-to-in-memory shape in `createRateLimiter` keeps local dev working without a Redis instance. The general lesson stuck harder than the specific tool: on serverless, any in-process state is per-invocation until proven otherwise.
- FCM web push with dual token storage: Tutors get notified on the web and, later, in the Flutter app, so tokens are stored as `fcmTokens.web` and `fcmTokens.app` on the profile document and both are sent to. Getting the service worker, the VAPID key, and permission prompts right on web push — and understanding that a token is per-device-per-browser, not per-user — was new.
- Reverse-engineering a schemaless model into a relational one: Writing `Docs/schema.md` meant reading every Firestore write path in `lib/utils/helpers/firebaseHelpers/*.ts` to work out what the implicit schema actually was, and that exercise surfaced problems no amount of reading the app would have: stat counters stored as strings, dates stored as ISO strings compared with `<=`, three dead collections (`user_roles`, `employees`, `messages`) with no live reads, and one `status` field meaning two different things depending on the collection.

### Takeaways

- Firestore security rules define what a client can write, not what your product means — anything with a quantity in it ("three applications a month") cannot live there, and pretending otherwise leaves the rule enforced only in code the user controls.
- "Avoid requiring composite indexes" is not a performance decision, it is a deferred bill; the index file is the cheap artifact and the full-collection scan is the expensive one.
- A manual payment approval step is not a smaller feature than a payment gateway — it adds a `pending` state that every screen has to render, and the states you forget are the ones support tickets are made of.
- Writing the target Postgres schema was the most effective audit of the Firebase app we ever ran, because a relational model refuses to compile the ambiguities a document store happily stores.
- `Promise.allSettled` over `Promise.all` for fan-out is not a style preference; with a third-party delivery channel in the loop, one bad token will otherwise silently cut off everyone after it.

### Journey

I came onto this as the second developer with the auth and notification surface as my half. That part started well — email verification, then 2FA with backup codes, then the notification system growing from in-app documents to email to FCM push over about a month of commits in November. Building each channel behind one `createNotification` call was the decision that held up best; by the time we added hire requests and subscription updates months later, wiring a new notification type into three channels was a single function call with a different `type` string. What I got wrong early was trusting the client. We called Firestore directly from React components everywhere because it was fast and gave us realtime for free, and I told myself the security rules were the boundary. They were, for row-level access. They were not for anything with business logic in it, and the seam shows in the codebase today: `applyToJob` in the client helper does the subscription check, the application-limit check, and the duplicate check, and `/api/tutor/apply-job` does the same three checks with a `TODO` where the actual write should be. Two enforcement points, one of them incomplete, both of them mine. That is exactly the kind of thing a client can be persuaded to skip. The moment it clicked was the June production audit. Seeing twenty-six findings on one page — a service account private key committed to the repo, notification endpoints anyone could read by changing a URL segment, a session cookie containing the string `"true"` — reframed the work from "ship features" to "the auth boundary is a thing that has to actually exist somewhere." I spent most of June and July on guards, `getVerifiedToken`, CSP headers, and rate limits rather than features, and the July cost work came out of the same shift: once I started reading our own query code adversarially, `getJobs` reading two thousand documents per page load was obvious and had been sitting there since November. What I would do differently is the shape, not the stack. Firebase Auth and FCM earned their place. Firestore-from-the-client did not — the reads it costs and the rules gymnastics it forces would both have been avoided by putting a thin server layer in front of the data from day one. That is essentially what the migration now under way does: Postgres on NeonDB behind a FastAPI service, with hire cascades, subscription expiry, and matching living in service-layer transactions instead of scattered across client helpers. Writing that schema, I got to name every ambiguity we had shipped, which is a strange and useful way to review ten months of your own work.

### Outcome

Shipped and running on Vercel. From the repository: 297 commits between 3 Oct 2025 and 23 Jul 2026, 76 page routes across four role dashboards plus public marketing and search, 15 API route handlers, and Firestore rules covering 16 collections with a default deny. Feature-complete for tutor, parent, company, and admin flows — job posting with soft delete, applications with shortlist/demo/hire/withdraw, employer-initiated hire requests gated on the tutor's plan, manual-payment subscriptions with monthly and yearly cycles, ratings, and admin moderation. The production audit (`Docs/PRODUCTION_AUDIT.md`) lists 26 findings; 23 are marked resolved, 3 remain open at medium severity. The cost pass moved every hot query to server-side filters with pagination and count aggregations, cut Sentry trace sampling from 100% to 10%, and reduced the notification listener from 50 to 20 documents. `Docs/COST_ESTIMATION.md` projects that work as ~$125–205/month down to ~$40–60/month at a 1,000-DAU baseline — a documented internal estimate against Firebase list pricing, not a measurement. Currently mid-migration: a 50-table PostgreSQL schema and a full REST API specification are written and committed, targeting NeonDB behind FastAPI, with 49 Dart entity models prepared for the Flutter client. The mobile app carries every role except admin, which stays web-only.

### Overview

Recrot is a full-stack job management platform connecting tutors, parents, and companies. Tutors can browse and apply to tutoring jobs, while parents and companies can post jobs and manage applications — all within a role-based, subscription-gated system. The platform features four user roles (Tutor, Parent, Company, Admin) with email-based 2FA, a complete job management lifecycle, a three-tier subscription system, a comprehensive admin dashboard, and Firebase-powered real-time notifications.

### Features

- Multi-Role Authentication — Four roles (Tutor, Parent, Company, Admin) with email 2FA and backup codes
- Job Management — Post, edit, and manage tutoring jobs with rich details
- Application Lifecycle — Submitted → Shortlisted → Hired / Rejected
- Subscription System — Three-tier plans (Free, Basic, Pro) with real-time enforcement
- Admin Dashboard — Full control over users, jobs, applications, subscriptions, and payments
- Push Notifications via FCM with per-device token management
- Email notifications for key events (password reset, application updates)
- Comprehensive Firestore security rules with document-level RBAC
- 101 reusable components organized by feature domain

### Stack

Next.js 15, TypeScript, Tailwind CSS v4, Radix UI, shadcn/ui, Framer Motion, Firebase (Firestore, Auth, Storage, FCM), Zustand, Nodemailer, jsPDF, Zod

## QuizNex

URL: https://muhammadsufiyanbaig.vercel.app/projects/personal/quiznex
Category: personal
Timeline: Mar 2026 – Jul 2026 (foundation in March, bulk of the build Jun–Jul 2026)
Role: Solo. I own every layer of this repo: the Postgres schema (23 tables, 16 enums, 7 generated migrations), all 71 API route handlers, the NextAuth v5 edge/node auth split, the browser-side proctoring engine, the Groq question-generation pipeline, the Safepay billing integration, and the AWS Amplify deployment. The git history has a single author. Parts of the security audit and some commit messages were drafted with AI assistance, but the design decisions, the debugging, and the shipping were mine.
Links: Live: https://quiznex.vercel.app/ | Code: https://github.com/muhammadsufiyanbaig/QuizNex

AI-powered proctored online examination SaaS for educational institutions — with in-browser TensorFlow.js eye tracking, Google Gemini question generation via LangChain, multi-role dashboards (Student / Teacher / Organization), passkeys, TOTP 2FA, and SSE real-time quiz control.

### Problem

A teacher running an online test has no idea whether the person answering is the person enrolled, or whether they have a second tab open. The workarounds are all bad: proctor over a video call and watch thirty tiles at once, force everyone into a physical room, or accept that the score measures search speed rather than knowledge. Teachers were also hand-writing every question, which is the slowest part of setting a quiz and the part they resent most. Schools that wanted to run this across many teachers had no way to see performance across classrooms without collecting spreadsheets by email.

### Constraints

One developer, no budget for a proctoring vendor or a GPU inference service — every detection model had to run in the student's browser on whatever laptop they own, including machines with no usable WebGL backend. Payments had to work in Pakistan, which rules out Stripe and forces Safepay, whose hosted-checkout docs and actual API disagree in several places. Deployment target was AWS Amplify SSR, which runs Next.js routes in Lambda and has its own opinions about which environment variable names it will pass through. And the users are teachers and students, not operators: anything that needs a manual step, a cron job, or a support ticket is a feature that will not get used.

### Architecture

It is a single Next.js App Router application — one deployable, no separate backend service.
That was a deliberate call. A separate API server would have bought clean separation and cost
me a second deploy target, a second set of secrets, and CORS on every call, for a project where
one person is on call. Instead, authorization lives in two tiers: `src/proxy.ts` runs at the
edge and does IP rate limiting plus a role-and-onboarding routing gate via an edge-safe
`auth.config.ts` (no Node imports, so it can run in the middleware runtime), while every
`/api/*` handler re-checks ownership against the database itself. The middleware explicitly
returns `true` for API paths rather than redirecting — a 302 to `/login` in response to a
`fetch()` is a terrible API contract. Persistence is Neon serverless Postgres through Drizzle,
chosen because a Lambda-per-request runtime cannot hold a connection pool, and because Drizzle's
generated SQL is inspectable when a query goes slow.

Real-time is the other place I rejected the obvious answer. Students need to see a quiz go live
and get auto-submitted when the teacher stops it. WebSockets on Amplify would have meant API
Gateway and a second piece of infrastructure; instead `/api/quizzes/[quizId]/stream` is a
Node-runtime SSE endpoint that polls the quiz row every 3 seconds and pushes only on change,
with a 15-second comment heartbeat to keep intermediaries from closing the pipe. It is
DB-polling wearing a push interface, and for "did this one row's status change" that is the
right amount of machinery. Scheduled quizzes follow the same philosophy: `autoActivateIfScheduled`
flips a `PUBLISHED` quiz to `ACTIVE` lazily on read, so there is no scheduler to operate.
Proctoring runs entirely client-side — TensorFlow.js and MediaPipe FaceMesh are dynamic imports
that do not load until camera permission is granted — and the server only ever receives
event rows, never video.

```flow
Student browser :: Fullscreen quiz session, camera, key/clipboard blocking (quiz-session-client.tsx)
FaceMesh in-page :: TF.js face landmarks every 1s; head-pose ratio decides gaze-away — no video leaves the device
proxy.ts (edge) :: IP rate limits per path class, role gate, onboarding redirects (role → 2FA → dashboard)
API route handlers :: Re-verify ownership per request, Zod-validate body, clamp client-supplied values
Attempt endpoints :: /answer saves per keystroke-debounce, /timer clamps to server elapsed, /events logs violations
Submit handler :: Drains in-flight saves, auto-scores MCQ, syncs totalMarks, flags, notifies student + teacher
Neon Postgres (Drizzle) :: Attempts, answers, proctoring_events, subscriptions — the audit trail a dispute needs
SSE stream route :: Polls quiz.status every 3s, pushes changes so a teacher stop auto-submits live sessions
```

### Challenges

- Gaze detection that failed every student in three seconds: The first version used iris landmarks (indices 468/473, available only with `refineLandmarks: true`) and incremented a violation counter on every 1-second tick while the ratio was out of range. Looking away once produced three violations in three seconds and ended the quiz. The fix was two-part: gate each episode behind a `gazeAwayRef` boolean with a `recordGazeBack()` that resets it, so one look-away counts once; and drop iris tracking entirely for a head-pose estimate — nose tip (kp 4) against the midpoint of the two cheeks (kp 234/454), flagging outside a 0.28–0.72 ratio. Head pose is a coarser signal than gaze, but it works without downloading the iris model, which is what made the model load usable on ordinary hardware.
- The model silently never loaded on some machines: The detector load was wrapped in a bare `.catch(() => {})`, so on any laptop where the WebGL backend failed to initialize, the camera preview looked perfectly normal and proctoring simply did nothing — no error, no signal, a completely unproctored exam that looked proctored. I now try `webgl` and explicitly fall back to `cpu`, log the failure, and paint an "AI scanning…" state onto the overlay canvas while `faceModelRef` is still null, so an unloaded model is visible rather than invisible.
- Last answer scored as zero: MCQ answers POST on click and Q&A answers POST on a 1500ms debounce, but submit reads answers straight from the database. A student who clicked an option and immediately hit submit raced their own save and lost the mark. I added a `pendingSavesRef` counter incremented before each save and decremented in `finally`, and `doSubmit` now spins on it with a 3-second deadline before calling the submit endpoint. A deadline rather than an unbounded wait — a hung request must not trap a student in a quiz that will not end.
- Two independent paths could activate one subscription: Safepay reports success twice — once via webhook, once via the client returning to the verify endpoint — and both called `activateSubscription`. Whichever ran second could also re-mark an already-succeeded payment as `FAILED`. The fix is an atomic claim: `UPDATE payments SET status='SUCCEEDED' WHERE id=? AND status='PENDING' RETURNING id`, and only the caller that gets a row back activates the plan. The same `onConflictDoNothing` + re-select pattern guards the org trial row, where two concurrent first requests would otherwise both try to insert.
- Amplify would not hand env vars to the Lambda: Five commits in one day chasing this. Amplify reserves the `AUTH_` prefix, so `AUTH_SECRET` never reached the runtime — renamed to `APP_SECRET`. Then the SSR Lambda turned out not to receive console-set variables at all, so `next.config.ts` now enumerates all 18 server variables under `env: {}` to bake them into the bundle at build time. That is a real tradeoff I accepted knowingly: secrets are now build-time inputs and rotating one requires a redeploy. There is still a leftover from that hunt — `/api/debug-env` is gone from the filesystem but remains listed in `PUBLIC_PATHS` in `auth.config.ts`.

### New ground covered

- MediaPipe FaceMesh / TensorFlow.js in the browser: I needed per-frame face landmarks with no server-side inference budget. What I understand now that I did not before: the landmark *set* you request determines model download size and load time far more than inference cost does, and a coarse signal that reliably loads beats a precise signal that silently fails to initialize.
- WebAuthn / passkeys via SimpleWebAuthn: Implementing registration and authentication end to end forced me to model the pieces properly — a `webauthn_challenges` table with 5-minute expiry because challenges are single-use server state, a signature `counter` per credential for clone detection, and a `passkey_tokens` one-time token to bridge a successful WebAuthn assertion into a NextAuth session, since NextAuth has no native passkey credential path.
- Safepay hosted checkout: My first non-Stripe payment gateway. The documented `/checkout/pay/` path is incompatible with v3 trackers; the working flow is create a tracker, mint a Time-Based Token from `/client/passport/v1/token` with an `x-sfpy-merchant-secret` header (not a bearer token), then hand both to `/embedded/`. The general lesson: with a smaller gateway, the SDK source is the specification and the docs are a summary of an older version.
- Server-Sent Events under a serverless SSR runtime: First time building a long-lived `ReadableStream` response with heartbeats and `req.signal` abort cleanup. I now understand why `X-Accel-Buffering: no` matters and why every `controller.enqueue` needs a try/catch — the client can vanish between two ticks of your own interval.

### Takeaways

- A proctoring failure that is invisible is worse than one that is loud: a silently unloaded detector produced exams that looked supervised and were not, which is the exact failure mode the feature exists to prevent.
- Any client-reported number that feeds a grade or an analytic has to be clamped server-side — the timer endpoint takes `min(client, server-computed elapsed)` because "I finished in 4 seconds" is otherwise a valid API call.
- Payment providers report success more than once by design, so idempotency belongs in a conditional `UPDATE … WHERE status='PENDING' RETURNING`, not in application-level checks that two Lambdas will pass simultaneously.
- Prompt injection in an education product arrives inside the uploaded coursework, not in the chat box — wrapping document text in `<document_content>` tags with an explicit treat-as-data-only anchor was a cheaper mitigation than any input filter I could write.
- Choosing a "boring" mechanism (poll-behind-SSE, lazy activation on read) removed two pieces of infrastructure I would otherwise have had to operate alone.

### Journey

I started this in March with the part I thought was the risk: authentication. Email plus OTP verification, Google OAuth, mandatory TOTP, passkeys, a role-selection gate. Three commits in and it was solid. Then the project sat almost untouched until June, and when I picked it back up I built the rest of the product — quizzes, attempts, analytics, notifications, admin — in a handful of very large commits. That gap is visible in the history and it is the honest shape of this project: an ambitious foundation, a pause, then a compressed sprint. The thing I got most wrong was believing proctoring was mostly a computer vision problem. I built iris tracking because it sounded like the correct answer, shipped it, and it was actively harmful — it ended honest students' quizzes within seconds of a single glance, and on machines where the model failed to initialize it protected nothing at all while displaying a camera feed that implied it did. The version that works is dumber on purpose. Head pose instead of gaze, one violation per episode instead of per tick, and a visible loading state so a broken detector is something you can see. I would rather over-tolerate a look-away than fail a student who looked at their own keyboard. The moment it clicked was the security audit in June. I went through the whole surface in one pass and found sixteen issues — cross-teacher S3 reads via unvalidated key prefixes, an SSE stream that never checked enrollment, OTPs in plaintext server logs, formula injection in CSV exports, a hardcoded fallback encryption key for TOTP secrets. None were exotic. All were the result of writing the happy path and moving on. Fixing them changed how I write handlers: every route now re-derives authorization from the database rather than trusting that middleware already handled it. Deployment was its own education. I lost the better part of a day to AWS Amplify quietly declining to pass `AUTH_SECRET` into the runtime because of a reserved prefix, and then to discovering the SSR Lambda does not receive console environment variables at all. The fix — baking every server variable into the bundle at build time — works, and I dislike it, and I have written down exactly why in the config so the next person does not think it was carelessness. What I would do differently: write tests. There are none in this repo, and the bugs that cost me the most — the gaze counter incrementing per tick, the submit/save race, the double subscription activation — are all cases a small integration suite would have caught in seconds instead of in production. I would also have kept the README honest as I went; it still describes Gemini via LangChain and iris-based gaze at a 10-events-per-60-seconds threshold, none of which is what the code does anymore.

### Outcome

Shipped and deployed on AWS Amplify (SSR) with Neon Postgres, with the last proctoring fixes landing 3 July 2026. What exists in the repo: 71 API route handlers, a 23-table schema across 7 migrations with 21 explicit indexes on hot paths, four roles (student, teacher, organization, admin) with a mandatory 2FA onboarding gate, browser-side proctoring producing durable `proctoring_events` rows, AI question generation from a topic or from PDF/DOCX/PPTX/TXT uploads via Groq (text and vision models, images extracted and passed alongside text), an admin console with an audit log, and a live Safepay billing flow with six plans priced in PKR (GOLD from 249/month through ORG_ENTERPRISE at 13,999/month) enforced by per-plan classroom and quiz limits. Two inconsistencies worth stating rather than hiding: `APP_CONFIG.proctoring` still declares a 10-violations-per-60-seconds threshold that no code reads — the session component hard-codes 3 violations — and `subscription.ts` creates a 15-day organization trial while the commit that introduced it describes 30 days. Both are documentation drift, not behaviour bugs, but both are the kind of thing that becomes a support argument.

### Overview

QuizNex is a full-stack SaaS examination platform supporting three distinct user roles — Students, Teachers, and Organizations — each with isolated dashboards, permissions, and feature sets. The proctoring system runs entirely in the browser: TensorFlow.js BlazeFace (478 facial landmarks) detects iris gaze direction at 30fps; 10+ gaze-away events in a 60-second rolling window auto-flags and force-submits the attempt. Teachers control quiz sessions via Server-Sent Events — starting or stopping a quiz propagates to all enrolled student sessions in real time without WebSocket infrastructure. AI question generation uses Google Gemini (Flash 1.5) via LangChain structured output with Zod schema enforcement; zero post-processing needed. Auth supports email+OTP, Google OAuth, TOTP 2FA, and Passkeys (SimpleWebAuthn FIDO2).

### Features

- In-browser TensorFlow.js eye tracking — 478 facial landmarks, 30fps inference, auto-flag on 10 violations/60s
- AI question generation — Google Gemini via LangChain structured output (Zod-validated, zero post-processing)
- Document-based generation — upload PDF/PPT/DOCX/TXT, generate contextual questions with multi-turn refinement
- SSE real-time quiz control — teacher starts/stops quiz; all student sessions update without WebSocket
- Multi-role auth — email OTP, Google OAuth, TOTP 2FA, Passkeys (FIDO2/WebAuthn) via SimpleWebAuthn
- Quiz lifecycle — DRAFT → PUBLISHED → ACTIVE → COMPLETED → ARCHIVED with scheduled activation
- Answer resilience — auto-saves every 30 seconds; localStorage fallback survives network drops
- Analytics — score distributions, per-question time heatmaps, student rankings, org-wide trends
- CSV (server) and PDF (client-side jsPDF, zero server cost) export for all analytics views

### Stack

Next.js 16, React 19, TypeScript, Tailwind CSS v4, Zustand, React Hook Form, Zod, Drizzle ORM, PostgreSQL (Neon), NextAuth v5, BCryptjs, SimpleWebAuthn, OTPAuth, Nodemailer, LangChain, LangGraph, Google Gemini, TensorFlow.js, BlazeFace, AWS S3, Cloudinary, Vercel

## Savvy

URL: https://muhammadsufiyanbaig.vercel.app/projects/personal/savvy
Category: personal
Timeline: Jan 2026 – Jun 2026
Role: Solo. I own everything in this repo: the seven-service backend (115 routed endpoints across `api-gateway`, `user-service`, `finance-service`, `bank-service`, `statement-analysis-service`, `ai-recommendation-service`, `notification-service`), the schema and Alembic migrations for the four PostgreSQL databases, the PostgreSQL row-level-security policies, the Kafka event contracts in `microservices/shared/events/`, the Next.js frontend (16 dashboard pages, 2 Zustand stores), the Docker Compose topology, the Kubernetes manifests and EKS bootstrap scripts, and the four GitHub Actions workflows. 264 Python files, ~22,250 lines of Python, 479 tests in 21 test files. No one else committed to it.
Links: Code: https://github.com/muhammadsufiyanbaig/savvy

Production-grade personal finance microservices platform — Claude AI bank statement analysis, 7-node LangGraph recommendation engine, Islamic finance features (Zakat/Qurbani/Shariah screening), Kafka event bus, 6 independent FastAPI services, and Kubernetes-ready Docker orchestration.

### Problem

Someone who wants to manage money according to Islamic obligation has to do it on paper or in a spreadsheet. Mainstream budgeting apps have no concept of zakat — you cannot ask them what 2.5% of your zakatable wealth is, because they do not know which of your assets are zakatable or what the nisab threshold is this year. They have no way to mark a loan as interest-bearing and treat that differently from an interest-free one. Planning for qurbani, hajj, or a cousin's wedding means keeping a separate goal-tracking sheet, then manually reconciling it against a bank statement at the end of the month. The reconciliation step is the expensive one: reading a PDF statement line by line and typing each transaction into a category is an hour of work that people simply stop doing after the second month.

### Constraints

Solo build, no budget for paid infrastructure during development, so everything had to run on a laptop via Docker Compose and still be deployable to a real cluster without a rewrite. Bank statements are the input and there is no aggregator API for Pakistani banks — statements arrive as PDF, CSV, or XLSX exports with wildly inconsistent formats, so parsing had to be tolerant rather than strict. The data is financial and personally identifying (CNIC numbers, IBANs, account numbers), which pushed encryption, PII log masking, and per-user database isolation from "later" into the first pass. And the AI path costs real money per call, so the design had to assume an adversarial user trying to burn API credits.

### Architecture

Seven FastAPI services, each with its own PostgreSQL database and its own Docker network. The obvious alternative was a modular monolith, which for a solo project would have been faster to build — I chose services anyway because the statement-analysis path has a completely different resource profile from CRUD: it is long-running, memory-heavy, and calls a paid external API, and I wanted to be able to scale and rate-limit that independently. The Kubernetes HPAs reflect this — `api-gateway` scales 2→8, `finance-service` and `bank-service` 2→6, `ai-recommendation-service` 1→4, all at 70% CPU target. The cost is real: `finance-service` cannot join against `bank-service` tables, so anything cross-domain goes through Kafka events rather than SQL.

Auth is centralised at the gateway and nowhere else. The gateway validates the JWT, injects `X-User-ID`, and signs the forwarded request with HMAC-SHA256 (`shared/utils/service_auth.py`, ±30s replay tolerance) so a downstream service can prove the request came from the gateway rather than from a compromised sibling container. Downstream, `finance-service` takes that header, sets `SET LOCAL app.user_id` on the transaction, and lets PostgreSQL RLS policies enforce ownership on all 11 user-owned tables — so a missing `WHERE user_id = ...` in application code returns zero rows instead of another user's expenses. Defence in depth, deliberately: the ORM filters, and the database refuses. Statement processing is fully asynchronous — the upload endpoint returns as soon as the file is in S3 and the event is published, because AI extraction of a 12-month PDF is not something to hold an HTTP connection open for.

```flow
Next.js client :: Uploads a PDF/CSV/XLSX statement to /api/v1/statements
api-gateway :: Validates JWT, injects X-User-ID, applies the 10-req/hour AI rate limit, HMAC-signs, proxies
bank-service :: Validates type and size, writes the file to S3, persists a BankStatement row, publishes bank_statement.uploaded
Kafka :: financial_bank_statement.uploaded, partitioned by user_id so one user's events stay ordered
statement-analysis-service :: Consumer thread downloads from S3, parses, sanitises the text, sends it to Claude with a cached system prompt
Categorisation pipeline :: ChromaDB vector match first, regex rules as fallback, confidence blended and written back as a learned pattern
Redis :: Holds live progress status the client polls while processing runs
finance-service + notification-service :: Consume statement.processed, create categorised expenses and an in-app/email notification
```

### Challenges

- Categorisation had to work when the AI was unavailable: My first pass sent every transaction to Claude for categorisation, which was slow and expensive per statement. I replaced it with a three-tier chain in `statement_processor._categorize_one`: ChromaDB similarity vote first (5 neighbours, cosine), regex rules if the vector store returns nothing, and a hard-coded `"Other"` fallback if both throw. Every categorised transaction is then written back into ChromaDB as a pattern, so the vector tier improves as it runs. The honest weakness is that the regex tier in `rule_categorizer.py` is US-merchant biased (Starbucks, Walgreens, Verizon) while the Claude prompt is tuned for Pakistani and Gulf merchants (Total Parco, Careem, LESCO, Easypaisa) — so the fallback path is worst exactly where the target user lives. That is the next thing I would fix.
- Prompt injection through an uploaded file: A bank statement is untrusted text that goes straight into an LLM prompt, and a PDF can carry invisible Unicode or HTML that a human reviewer never sees. `app/ai/input_sanitizer.py` strips control and zero-width characters, strips HTML/XML tags, truncates to 50,000 chars, and rejects the document outright on 13 injection patterns. I paired that with an anti-disclosure clause in the system prompt itself — if the model detects an override attempt it returns `[]` rather than prose. Belt and braces, because regex alone will not catch a paraphrase.
- Refresh-token theft detection without a session table: Rotating refresh tokens is easy; detecting that a stolen one was replayed is not. `rotate_refresh_token` stores each token's `jti` in Redis and deletes it on use — if a valid JWT arrives whose `jti` is already gone, the token was used twice, which means theft, and it raises rather than issuing new credentials. The same Redis sorted set enforces a 5-session cap per user and evicts the oldest. The tradeoff I accepted: if Redis is down the rotation fails open and issues tokens anyway, because locking every user out of the product when a cache dies is the worse failure.
- Kafka consumers that lose work on crash: The shared `EventConsumer` in `shared/utils/kafka_client.py` sets `enable_auto_commit=False` and only commits after the handler returns, so a crash reprocesses the message. The statement-analysis consumer in `app/events/consumer.py` builds its own `KafkaConsumer` without that flag, so it auto-commits — a container restart mid-extraction silently drops that statement, and the user sees a job stuck at 50%. I know about it, the fix is to route it through the shared client, and it is not done.
- RLS versus Alembic: Turning on `FORCE ROW LEVEL SECURITY` broke migrations, because Alembic runs outside any request and has no `app.user_id` set. The fix was `current_setting('app.user_id', true)` with the missing-ok flag so it returns NULL instead of raising, plus a documented `BYPASSRLS` admin role for data migrations. Getting `SET LOCAL` rather than `SET` was the other half — with a connection pool, a session-scoped variable leaks one user's ID into the next request on the same connection.

### New ground covered

- PostgreSQL row-level security: I had always treated ownership checks as application logic. Writing `database/rls/finance_rls.sql` forced me to understand that RLS is not a `WHERE` clause helper — it changes what `UPDATE` and `DELETE` silently no-op on, interacts badly with connection pooling unless you use `SET LOCAL`, and needs an explicit bypass path for migrations. It is the only control in this system that still works if my ORM code is wrong.
- LangGraph: Used in `ai-recommendation-service/app/workflows/` to model recommendation generation as an explicit state graph — profile lookup, AI generation, rule-based fallback, merge — rather than nested try/except. What I understand now that I did not before: the value is not orchestration (four sequential nodes is not hard), it is that the fallback becomes a node in the graph instead of an exception handler, so "AI returned one weak suggestion" and "AI threw" are the same code path.
- Anthropic prompt caching: The extraction system prompt in `app/ai/prompts.py` is deliberately long — full category taxonomy, merchant-cleaning rules, 12 worked examples — because it must clear the 1024-token threshold for `cache_control: ephemeral` to engage. Learning that a *bigger* static prompt plus a tiny dynamic user turn is cheaper than a compact prompt rebuilt every call inverted how I write prompts.
- ChromaDB as a learning cache, not a search index: I expected to use a vector store for retrieval. Here it is a write-back memory: every categorised transaction becomes a labelled example, so the system's accuracy on one user's recurring merchants improves without retraining anything.

### Takeaways

- Fail-open and fail-closed are per-control decisions, not a system-wide policy — rate limiting fails open because availability matters more than a perfect limit, while the HMAC check fails closed in production because a forged internal request is worse than an outage.
- Designing the AI rate limit (10 requests/hour, separate key namespace) before writing the AI feature was the single highest-leverage decision, because cost-exhaustion is the one attack that hurts before you notice it.
- Splitting into services made every cross-domain read an event, and I underestimated how much of the codebase would end up being producer/consumer plumbing rather than product logic.
- Documentation and code drift apart faster than either drifts from reality — the README quotes nisab as 87.48g gold / 612.36g silver while `zakat_service.py` uses 85g / 595g, and nothing caught it because no test asserts the constants.
- A "learned pattern" store is only as good as its write path: because ChromaDB writes happen sequentially after a parallel categorisation pass, that loop is the throughput ceiling of the whole pipeline, and I did not see it until I profiled.

### Journey

It started as a specific irritation rather than a product idea. I wanted to plan for zakat, qurbani, and a set of upcoming family expenses in the same place I tracked ordinary spending, and every app I tried treated the religious obligations as if they did not exist. The original brief I wrote for myself is still in the repo as `Project Brief.txt` — it is one long paragraph, half feature list and half wishful thinking, and reading it now the thing that stands out is how much of it survived. I got the AI part backwards at first. My assumption was that Claude would do the categorisation and the deterministic code would just be glue. That is expensive and it is fragile — one API outage and the core feature returns nothing. The rewrite inverted it: deterministic categorisation is the default path, the vector store makes it better over time, and the model is used for the one thing rules genuinely cannot do, which is turning a wall of ragged PDF text into structured rows. The confidence score in `confidence_scorer.combine` exists because of that inversion — I needed a single number that says how much to trust a row regardless of which tier produced it. The moment it clicked was writing the RLS policies. Up to that point security was a checklist I was working through. Running a query with the wrong `app.user_id` and watching PostgreSQL return zero rows for data I knew was there reframed it: the question stopped being "did I remember the filter" and became "what still holds when I forget". Everything after that — HMAC between services, the six-network topology where database containers are unreachable by non-owning services, the PII masking filter attached at the logging layer so no future `logger.info` can leak a CNIC — came out of that same question. What I would do differently: I would not have committed this as two commits. The whole build happened locally and landed in one drop, which means the git history records nothing about where the difficulty actually was — the three rewrites of the categorisation pipeline, the RLS-versus-Alembic fight, the switch from `SET` to `SET LOCAL`. All of that is reconstructable from the code and the comments I left, but it should have been in the log. I would also have written the integration tests earlier. 479 unit tests is a reasonable number, and they are concentrated exactly where I was least confident — 46 in the recommendation service, 43 on gateway auth, 33 on categorisation — but they all mock Kafka, Redis, S3, and Claude, which is why a live bug like the health-score endpoint querying `ZakatRecord.calculated_at` and `is_paid` when the model actually has `calculation_date` and `payment_status` sat there undetected. The unit tests cannot see it. A single end-to-end run would have.

### Outcome

Module 1 shipped: 115 routed API endpoints across 7 services, 4 PostgreSQL databases with Alembic migrations, RLS on all 11 user-owned finance tables, a Kafka event bus with 5 typed event families, ChromaDB-backed categorisation, and a Next.js frontend covering 16 dashboard pages including zakat, qurbani, sadaqah, hajj/umrah, net worth, and the Islamic financial health score. 479 tests run in CI across a 7-service matrix, with `pip-audit`, `bandit`, `npm audit`, and Trivy image scanning on every push plus a weekly CVE sweep. Not shipped: production deployment. The EKS manifests, HPAs, cert-manager issuer, external-secrets bindings, and the 11 numbered bootstrap scripts are written and the repo's own status section marks it as pending cloud provisioning — nothing has run against a real cluster. The four security items the catalogue marks partial or deferred (K8s Secrets among them) are still open.

### Overview

Savvy is a distributed, event-driven financial management system with 6 independent FastAPI microservices (User, Finance, Bank, Statement Analysis, AI Recommendation, Notification), each with a dedicated PostgreSQL database. Services communicate exclusively via Kafka (20+ event types) — no synchronous inter-service HTTP. The Statement Analysis service uses Claude AI to extract and categorize every transaction from uploaded PDF/CSV/Excel bank statements with confidence scoring. The AI Recommendation service runs a 7-node LangGraph workflow: Expense Pattern Analysis → Income Assessment → Goal Gap Analysis → Market Research (yfinance) → Shariah Compliance Screening → Risk Assessment → Claude synthesis. Islamic finance features include a full Zakat calculator (Nisab threshold + 2.5% on zakatable wealth), Qurbani savings goals, and Shariah-compliant investment filtering. The frontend is a Next.js glassmorphism UI with Framer Motion and Recharts.

### Features

- Claude AI bank statement analysis — PDF/CSV/Excel upload → S3 → extract transactions → categorize with confidence score
- 7-node LangGraph recommendation workflow — expense analysis, income assessment, goal gap, market research, Shariah screening, risk assessment, Claude synthesis
- Islamic finance — Zakat calculator (Nisab + 2.5% on zakatable wealth), Qurbani savings, Shariah-compliant investment filter
- 6 independent microservices — each with dedicated PostgreSQL database, no shared DB
- Kafka event bus (20+ event types) — no synchronous inter-service HTTP calls
- Budget enforcement via Kafka events — alert on threshold, notification on overspend
- Recurring expenses — auto-created monthly via scheduler
- 16+ expense categories — including Zakat, Qurbani, Eid, Emergency
- Kubernetes manifests + Docker Compose (~15 containers) for local and production orchestration

### Stack

Next.js, TypeScript, Tailwind CSS, Framer Motion, Recharts, Zustand, FastAPI, Python, SQLAlchemy 2.0, Alembic, Pydantic v2, Apache Kafka, Redis, PostgreSQL ×4, AWS S3, Anthropic Claude, LangGraph, LangChain, ChromaDB, yfinance, OneSignal, Docker, Kubernetes

## One Developer Investor ERP

URL: https://muhammadsufiyanbaig.vercel.app/projects/client/one-developer-investor-erp
Category: client
Built for: Innovative Widget
Timeline: Jul 21 – Jul 25, 2026 (5-day build, single developer)
Role: Solo developer on this repo, end to end: architecture, all 44 route/API files, the ResERP integration layer (`lib/reserp-client.ts`, `lib/session.ts`, `lib/units.ts`, `lib/schedules.ts`, `lib/receipts.ts`, `lib/pdf.ts`), the auth flow, and the UI system (11 shared components, 4-state pattern on every route). I also wrote and maintained the five planning docs the repo is built from (`docs/PRD.md`, `docs/API_Integration.md`, `docs/Frontend_Routes.md`, `docs/Design.md`, `docs/Implementation_Plan.md`) and updated them in place as live testing against the real ResERP API contradicted what the vendor's Postman collection and PDF documentation said. This sits alongside two earlier repos I also built for the same client — `backend_onecapitalinvestment` (Node/Express + Postgres, 25-table schema, manual admin data entry) and `onecapitalinvestment` (the Next.js admin/investor dashboard on top of it) — which this project replaces for the buyer-facing side once the client's ERP access came through.
Links: Live: https://onecapitalinvestment.vercel.app/

Full-stack property investment management platform enabling administrators to manage Rental, Development, and Holding properties with a draft/publish workflow, comprehensive audit logging, and a dedicated investor portal with portfolio analytics.

### Problem

Square One Capital sells property units to buyers who then pay in installments over time. Before this portal, a buyer with a question about their account — how much is due, when, whether last month's cheque cleared, or getting a copy of their allotment certificate for a bank loan application — had no self-service option. They called or visited the office, and staff looked it up manually in the ERP (ResERP) or the company's own records. The earlier version of this client's system (the `onecapitalinvestment`/`backend_onecapitalinvestment` repos) tried to solve this by having Square One Capital's own admins manually re-key investor, property, and transaction data into a separate Postgres database — which meant two sources of truth and admin time spent on data entry that the ERP already had.

### Constraints

ResERP is the client's existing, external, staff-facing ERP — not something this project could modify. Its client-facing API surface is fixed at 11 endpoints, all read-only except login, with no admin/bulk endpoint, no pagination, and (per `docs/API_Integration.md` §6) no documented error-response shape or shape consistency between endpoints. The vendor-supplied documentation (Postman collection + PDF) turned out to be wrong or incomplete on two load-bearing points — the `PDF_TOKEN` auth and the `/product/:id` response shape — discovered only by making real calls against the live API. Solo developer, five-day build, no dedicated QA, and (per `docs/Git_Workflow.md` §4) an open question the whole build shipped under: whether a sandbox ResERP instance exists at all, or whether `dev`/`staging` testing during the build was hitting real buyer CNIC/password data.

### Architecture

The core decision was to build nothing but a frontend. ResERP already holds every fact this app needs — units, payment schedules, receipts, PDFs — so there is no database, no data model, no sync job. The app is a Next.js BFF (backend-for-frontend): every ResERP call happens server-side in a Route Handler or Server Component, using a `RESERP_BASE_URL` env var and a bearer token that never reaches the browser. This was chosen over a client-side SPA calling ResERP directly because that would have put the ResERP token in browser JS and made CNIC/password traffic visible in devtools — unacceptable for a portal whose only credential is a national ID number. Auth uses a two-tier check: `proxy.ts` (Next.js 16's replacement for `middleware.ts`) does an optimistic cookie-presence check to redirect unauthenticated requests before they render anything, but every Route Handler independently re-verifies the session via `verifySession()` before touching ResERP — the proxy check alone is documented upstream as insufficient, and I treated that as a hard rule rather than a suggestion.

The session itself is a signed, httpOnly JWT (via `jose`) holding the ResERP login token, not the token in `localStorage` or a readable cookie. Because ResERP has no admin/list-all endpoint, there is no server-side aggregation or caching layer beyond what a single buyer's own session token can see — this rules out ever building an admin view against this API without a second, different ResERP credential the client would have to provide. Data fetching is organized as one `lib/*.ts` module per ResERP resource (`units.ts`, `schedules.ts`, `receipts.ts`, `pdf.ts`), each owning its own response typing rather than sharing one generic envelope type, because `docs/API_Integration.md` §6 documents that ResERP's `data` shape is inconsistent across endpoints — sometimes a named key, sometimes a bare array, sometimes an object that becomes an array only in list mode. Typing each one separately, instead of one clever generic wrapper, is what caught the bugs below at the type level once they were understood.

```flow
Buyer :: Logs in with CNIC + password on /login, mobile-first, no ResERP knowledge needed
proxy.ts :: Optimistic redirect — cookie present? else send to /login (not the security boundary)
Route Handler / Server Component :: verifySession() re-checks the JWT, then calls lib/reserp-client.ts with the decrypted token
lib/reserp-client.ts :: Single reserpFetch() wrapper — attaches bearer token, normalizes success:false and the 204 "no data" case into typed errors
ResERP API (external) :: onedev.myreserp.com/api — source of truth for units, schedules, receipts, PDFs; read-only except /login
lib/units.ts / schedules.ts / receipts.ts / pdf.ts :: Per-resource typing so list-vs-single shape differences don't get silently unified
Server Component (page.tsx) :: Renders one of loading/empty/error/unauthorized/data — never assumes happy path
Browser :: Receives rendered HTML/PDF stream only — never sees RESERP_BASE_URL, the bearer token, or raw ResERP JSON
```

### Challenges

- Vendor docs vs. live behavior on `PDF_TOKEN`: The Postman collection specified a separate static `PDF_TOKEN` for all five PDF endpoints, and the client handed over a value for it. A live curl test against `/pdf/statement/:sell_id` with that token returned `HTTP 200 {"success":false,"code":"401","message":"Token does not match"}` — a 200 status hiding a failure. Retesting the same five endpoints with the regular per-user login token instead returned real PDFs every time. I rewrote `lib/pdf.ts` to use the login token everywhere and documented the discrepancy in `docs/API_Integration.md` §0/§5 so it wouldn't get "fixed" back to the wrong behavior later by someone trusting the collection.
- A response shape bug that broke every unit page in production: `GET /product/:id` (single unit) does not return the same shape as `GET /product` (list) — list mode nests an array under `data.product`, single mode nests a bare object. `lib/units.ts` originally assumed both were arrays and read `data.product[0]`, which is `undefined` on a plain object. Every single buyer, on every unit, hit `notFound()`. Fixed in commit `421c254` by splitting `ProductListData`/`ProductSingleData` into distinct types instead of one shared `ProductsData` — the same mistake was almost repeated on `/schedules`, which I caught by explicitly re-verifying that endpoint's list-mode shape live rather than assuming parity a second time.
- 204 isn't an error, it's "zero rows": ResERP signals an empty result as HTTP 200 with `success:false` and `code:204` ("No Data Found"), not an empty array/object. The generic `reserpFetch()` throws on any `success:false`, which would have turned "buyer owns zero units" into a rendered error page instead of an empty state. `isNoDataError()` in `lib/reserp-client.ts` exists specifically to unwrap that one code back into a valid empty result at the call site, while every other error code still fails loudly.
- Dark-mode cards were invisible: `Card` was styled `bg-background`, and in dark mode `--background` and the card's border-based elevation made cards visually indistinguishable from the page — the only cue was a 12%-opacity border. Fixed by introducing a separate `--surface` token (`#161616` vs. `#0a0a0a` background) so elevated surfaces are actually a different color, not just a different border, in dark mode — documented in `docs/Design.md` §2 specifically so future elevated UI (modals, dropdowns) doesn't repeat it.
- A one-character case bug broke the logo on Vercel: The header referenced `/square.png` but the tracked asset was `public/square.PNG`. It worked locally on a case-insensitive filesystem and 404'd on Vercel's case-sensitive one. Two commits (`9577a10`, `479c9ac`) — first renaming the asset, then finding a second reference that still pointed at the old casing — a small reminder that "works on my machine" for static assets is a filesystem property, not a code property.

### New ground covered

- Next.js 16's `proxy.ts`: This project's first exposure to Next 16 replacing `middleware.ts` with a root-level `proxy.ts` export. I couldn't rely on training-data assumptions about Next middleware here and had to verify behavior against `node_modules/next/dist/docs/` directly before writing the auth guard, which is now the documented practice for anything in this codebase where Next 16 semantics might have changed.
- BFF pattern with `server-only` as an enforced boundary: Using the `server-only` package to make it a build error (not just a convention) if `lib/reserp-client.ts` or `lib/session.ts` were ever imported into a Client Component — turning "never expose the token" from a code-review rule into something the compiler catches.
- Treating a third-party API's documentation as a hypothesis, not a contract: Every "resolved" open question in `docs/PRD.md` §9 and `docs/API_Integration.md` reflects a live call made specifically to confirm or refute what the vendor's Postman collection claimed. That habit — verify the documented shape against a real request before writing the type — is what caught the `/product/:id` bug before a second occurrence on `/schedules`.

### Takeaways

- A `200 OK` with `success:false` in the body is still a failure state that needs explicit handling — HTTP status alone is not a reliable signal against this API.
- List and single-item response shapes from the same endpoint family should never be assumed to match without checking both directly; the cost of being wrong here is a silent, total outage of a whole page, not a type error.
- A vendor's static, documented credential (`PDF_TOKEN`) can simply not work in production while returning a 200 status that looks like success at a glance — trust the live response, not the collection.
- Elevation in a dark theme needs a distinct surface color, not just a border — border-only elevation reads as "flat" the moment background and surface share a hex value.
- Filesystem case-sensitivity differences between local dev and the deploy target (macOS/Windows vs. Vercel's Linux-based build) are a real, recurring class of "works locally, breaks in prod" bug for static asset references.

### Journey

This started as a pivot, not a greenfield idea — the client already had `onecapitalinvestment` and `backend_onecapitalinvestment` running, a full custom platform where Square One Capital's admins manually entered every investor, property, and transaction. When ERP (ResERP) API access became available, the obvious question was whether to retrofit the existing Postgres-backed app to sync from ResERP, or build something new and much smaller that just reads from ResERP directly. I chose the second option, and this repo is the result — no database at all, because ResERP already is the database for everything this portal needs to show. The first real scare was two days in, seeing every unit detail page return "not found" for every account in production. My first instinct was to suspect the auth layer, since that was the newest code — I spent time re-checking `verifySession()` and the cookie flow before actually diffing the single-unit response against the list response and seeing the object-vs-array mismatch. That was a useful, humbling lesson: when something fails for literally every user, look at the data shape before the auth code, because "works for nobody" points at something structural, not something session-specific. The `PDF_TOKEN` question is the one I'm most glad I didn't guess on. The client had handed over what looked like a legitimate, documented credential, and it would have been easy to wire it in, watch the PDF downloads return something, and move on without noticing the response was a 200-status JSON error body pretending to be a success. Actually curling it and reading the bytes back — `%PDF-1.7` vs. a JSON error — is the difference between shipping a working feature and shipping a broken one that looks fine in a quick manual test. If I were doing this again, I'd push harder, earlier, on the open sandbox-environment question flagged in `docs/Git_Workflow.md` §4 — building and testing against what might be the live production ResERP instance with real buyer CNICs is a real data-handling risk that got documented rather than resolved, because resolving it wasn't in my control. I'd also add automated tests around the response-shape assumptions in `lib/units.ts` and `lib/schedules.ts` specifically, since both bugs found so far in this project were exactly that category of mistake, and there's currently no test suite catching a regression to the same pattern.

### Outcome

Shipped through six implementation phases (`e900197` initial foundation through the `feature/phase-6-polish` merge) between 2026-07-21 and 2026-07-25, plus a further round of dark-theme redesign and bug fixes on 2026-07-25, promoted through the `dev → staging → main` branch flow described in `docs/Git_Workflow.md`. Delivered: CNIC-based login, a buyer's full unit list and per-unit detail (size, layout, registration number, tenant status, utility-bill counters), payment schedule with status filtering, upcoming-installment highlight, receipt history, and all five ResERP PDF documents (statement, booking form, allotment certificate, receipt, upcoming-installment notice) as authenticated streaming downloads — with the `PDF_TOKEN` and `/product/:id` issues fixed before or shortly after they reached production. md` §8 — none of these are recorded in the repo.

### Overview

One Capital Investment is a full-stack property investment management web application. Administrators manage four property types (Rental, Development, Holding, Other) through a draft/publish workflow, track investor lifecycles, run support ticket queues, and review a comprehensive audit log that records every create/update/delete with IP and user agent. The investor portal lets users browse available properties, view their portfolio and transaction history, upload documents, and review personal analytics. Built with Next.js 15 / React 19 on the frontend and a Node.js + Express.js REST API on the backend, backed by PostgreSQL (Neon) via Drizzle ORM. Redis caching with configurable TTL and graceful degradation, Cloudinary for organized media storage, and Nodemailer for OTP-based email flows round out the stack.

### Features

- Full property management with draft/publish workflow (Rental, Development, Holding)
- Investor lifecycle management — approve, disable, edit investor accounts
- Investment & transaction tracking across all property types
- Support ticket system with priority levels and team assignment
- Advanced analytics dashboard with charts and trends
- Comprehensive audit log (tracks every create/update/delete with IP and user agent)
- Investor Portfolio — Browse properties, investment history, personal analytics
- OTP-based password reset via email
- Redis caching with TTL strategies and graceful degradation

### Stack

Next.js 15, React 19, TypeScript, Tailwind CSS v4, Node.js, Express.js, PostgreSQL (Neon), Drizzle ORM, JWT, Zustand, Recharts, Redis, Nodemailer, Cloudinary

## Schema Genie

URL: https://muhammadsufiyanbaig.vercel.app/projects/package/schema-genie
Category: package
Timeline: Mar 2026 (git history spans 7–8 Mar 2026).
Role: Solo. I own everything in the repo: the six-stage inference pipeline (type detection, cardinality, relationship detection, fact-table selection, schema-type recommendation, DDL generation), the four warehouse DDL generators behind a common `BaseDDLGenerator` interface, the dataclass IR (`ColumnDefinition` / `ForeignKey` / `TableDefinition` / `InferredSchema`), the deploy and diagram side-paths, the 68-test suite, and the PEP 621 packaging with four optional connector extras. There is no other contributor and no reviewer.
Links: Code: https://github.com/muhammadsufiyanbaig/schema-genie | PyPI: https://pypi.org/project/schema-genie/

Automatically infers optimal star and snowflake schemas from raw DataFrames and generates production-ready DDL for Snowflake, Redshift, BigQuery, and PostgreSQL in seconds.

### Problem

Every time a team lands a new dataset in a warehouse, someone has to sit down and decide which columns are measures, which are dimensions, which table is the fact table, where the foreign keys are, and whether the dimensions should stay flat or be normalised. That work is repetitive and almost entirely mechanical, but it blocks everything downstream — nobody can write a query or train a model until the tables exist. The usual coping strategy is a data engineer hand-writing `CREATE TABLE` statements from a spreadsheet of column names, then discovering three weeks in that the dimension tables have no history columns and point-in-time joins are impossible.

### Constraints

It had to work on data as it actually arrives — a pandas DataFrame or a CSV with no schema file, no dictionary, no annotations. Nothing but column names, dtypes and values to reason from, which rules out anything that needs metadata and forces heuristics. It had to emit DDL for four warehouses whose dialects disagree on surrogate keys, type names, constraint support and physical layout, without asking the caller to know any of that. And it had to be a library other people install, so the hard dependency list needed to stay small (pandas, numpy, sqlglot, pyyaml) with connectors pushed to optional extras — a user targeting Postgres should not be made to install the Snowflake connector.

### Architecture

The centre of the design is a warehouse-neutral intermediate representation. Inference never produces SQL; it produces dataclasses — a `TableDefinition` holding `ColumnDefinition`s tagged with one of six *semantic* types (`id`, `date`, `currency`, `measure`, `category`, `text`) plus `ForeignKey` edges. Only the generator layer knows SQL exists. The alternative was to let each stage emit dialect-aware fragments and concatenate them, which would have been fewer lines but would have meant every new warehouse touched every inference module. With the IR, adding a target means one file: a `TYPE_MAP` dict and a `_table_ddl` method on a `BaseDDLGenerator` subclass. `SchemaGenie.__init__` resolves the target through a `GENERATORS` registry and fails loudly on an unknown name rather than falling back to a default dialect.

The semantic-type layer is the other deliberate choice. It would have been simpler to map pandas dtypes straight to SQL types, but dtypes cannot tell you that `revenue` should be `NUMBER(18,2)` and not `FLOAT`, or that a three-value string column is a category rather than free text. So detection runs name regex → dtype → cardinality ratio, in that order, first match wins, and every downstream decision keys off the semantic type instead of the dtype. That single indirection is what lets the same inferred schema produce `NUMBER(18,2)` on Snowflake, `DECIMAL(18,2)` on Redshift and `NUMERIC` on BigQuery, and it is what makes the fact-table scorer possible at all — the score is a function of how many columns are measures, which is a semantic question.

```flow
Caller :: pandas DataFrames or CSVs handed to SchemaGenie.infer / infer_multi
detect_all_types :: name regex, then dtype, then cardinality ratio (default 0.05) -> six semantic types
detect_relationships :: name-stem pre-filter, then |A n B| / min(|A|,|B|) >= 0.8 -> FK edges
select_fact_table :: (measures/cols) + (id_cols * 0.2) + log10(rows)/10; highest score becomes the fact
recommend_schema_type :: any dimension with >40% category columns -> "snowflake", else "star"
_build_table :: ColumnDefinition / TableDefinition / ForeignKey — the warehouse-neutral IR
BaseDDLGenerator subclass :: per-target TYPE_MAP plus dialect physical design -> DDL string
export_ddl / deploy_ddl :: write a .sql file, or execute against a live connection
```

### Challenges

- Picking the fact table without a schema: Row count alone is wrong — a date dimension can be longer than a small fact table, and a wide lookup table can be huge. I settled on an additive score in `score_table`: measure density (`measures/total_cols`), an FK-candidate bonus (`id_count * 0.2`), and `log10(rows)/10` so size influences the result without dominating it. The log damping is the part that matters; with raw row count in the sum, the largest table always won regardless of shape.
- Relationship detection is quadratic twice over: Comparing every column of every table against every column of every other table is O(tables² × cols²), and each comparison builds two Python sets over the whole column. The fix was a cheap name-stem pre-filter in `_columns_are_related` so only plausibly-matching pairs get the set intersection. The prefilter itself has a latent bug I have not fixed: it uses `str.rstrip("_id")`, which strips a *character set*, not a suffix — `"paid".rstrip("_id")` returns `"pa"`. It has not produced a false match on any test data yet, but it is wrong and should be a proper suffix strip.
- The dialects refuse to stay behind one abstraction: `BaseDDLGenerator` gives each target a `TYPE_MAP` and a `generate` method, and that covers types cleanly. Physical design does not abstract at all: Redshift wants `DISTKEY`/`SORTKEY` appended after the closing paren, BigQuery wants `PARTITION BY DATE(col)` and rejects foreign-key constraints outright, Postgres wants separate `CREATE INDEX` statements after the table. I stopped trying to generalise it and let each `_table_ddl` be openly dialect-specific — the shared parts are the type map and the SCD audit columns, and that is where the abstraction ends. The visible cost is that BigQuery nullability is currently smuggled into `OPTIONS(description='NOT NULL')` instead of being a real constraint.
- Dead branches that heuristic ordering creates: In `detect_type`, the boolean check sits after `is_numeric_dtype`, and pandas reports bool columns as numeric — so the bool branch never runs, and bool columns fall through to the cardinality test (which does classify them as `category`, so the output is right by accident). The same function computes `null_ratio` and never uses it. Ordered first-match-wins heuristics are readable but they make unreachable code easy to write and hard to notice; the tests all passed because they only ever asserted the final label.
- Multi-statement DDL against four different drivers: `deploy_ddl` has to execute a single DDL string, but the drivers disagree — psycopg2 takes the whole blob, Snowflake and BigQuery need one statement per call. I split on `;` in `_split_statements`, which is correct for generated DDL (no string literals, no procedural bodies) and would break immediately on hand-edited SQL containing a semicolon inside a quoted default. It is a documented-by-code assumption, not a general SQL splitter, and this whole module has no test coverage.

### New ground covered

- SCD Type 2 as a generation concern: I had treated slowly-changing dimensions as an ETL problem — something you handle when writing merge logic. Putting `_valid_from` / `_valid_to` / `_is_current` on every dimension table at DDL time inverts that: the history columns exist before the first row lands, so point-in-time joins are possible from day one instead of requiring a migration months later. What I understand now is that the expensive part of SCD2 is never the merge query, it is retrofitting the columns onto a table that already has data.
- Warehouse physical design as DDL, not tuning: Before this I thought of `DISTKEY`, `SORTKEY` and `PARTITION BY` as things you add later when queries get slow. They are `CREATE TABLE` decisions — Redshift distribution in particular is painful to change after load. Building the generators forced me to learn what each warehouse actually keys off, and why BigQuery's partitioning story (prune by date column) is a completely different mechanism from Redshift's (co-locate rows by join key).
- Library packaging with optional extras: First time building a `pyproject.toml` where the dependency graph is a real design decision rather than a formality — four `[project.optional-dependencies]` groups (`snowflake`, `redshift`, `bigquery`, `diagram`) with lazy imports inside `deploy.py` and `diagram.py` that raise an `ImportError` naming the exact `pip install schema-genie[...]` command. The lazy-import-plus-helpful-error pattern is now my default for any optional integration.
- Graphviz HTML-like table labels: `export_diagram` builds `<TABLE>` markup as node labels to render ER boxes with per-column rows and colour-coded headers. Not difficult, but I did not know Graphviz supported it at all, and it turned a would-be custom renderer into about forty lines.

### Takeaways

- Introducing a semantic type layer between pandas dtypes and SQL types was the single decision that made four dialects cheap, because every dialect difference collapsed into one dict lookup.
- Additive heuristic scores need their terms damped to comparable ranges — the `log10(rows)/10` term exists purely so table size cannot outvote table shape.
- First-match-wins heuristic chains read well and hide unreachable branches; the bool branch in `detect_type` was dead for weeks and every test still passed because assertions checked the label, not the path.
- Tests clustered where I was least confident — the four DDL generators carry roughly a third of the 68 tests — and the modules with zero dedicated tests (`deploy.py`, `diagram.py`, `relationships.py`, `cardinality.py`, `scd.py`) are exactly the ones where I later found the loose assumptions.
- A README that quantifies impact with industry-average rates rather than measured runs is a liability in any conversation with an engineer; estimates should be labelled as estimates in the artifact itself, not just in the caveat line.

### Journey

I started this because I had written the same `CREATE TABLE` block by hand too many times, and the third time I typed `_valid_from TIMESTAMP` I realised the whole thing was a function of the DataFrame I already had open. The first version was much smaller than what shipped — type detection and a single Snowflake generator, no relationships, no fact selection. It worked on one table and was useless on real projects, because a real project is five tables and the interesting question is which one is the fact. The thing I got wrong early was believing the fact table was the biggest table. That is true often enough to be seductive and wrong often enough to be dangerous — a date dimension is the obvious counterexample and it broke the first version immediately. Rewriting `select_fact_table` around measure density, with row count damped through a log, is when the inference stopped feeling like a guess. The related mistake was letting relationship detection compare every column against every column; on anything wider than a toy dataset it crawled, and the name-stem prefilter is a patch on a design I would think about differently now. The moment it clicked was writing the second generator. I had built Snowflake first with the SQL strings inline, and porting it to Redshift meant either duplicating the whole thing or extracting an IR. Extracting the IR took an afternoon and the third and fourth generators each took under an hour — which is the clearest evidence I have that the abstraction boundary was in the right place. It also told me exactly where the boundary is *not*: physical design (distribution, partitioning, indexes) does not generalise, and I stopped fighting it. What I would do differently: I let the repo drift in ways that are small but embarrassing. `sqlglot>=10.0` is a hard dependency that nothing in the package imports — it was in the plan for SQL validation and never got wired in, but it still forces itself on every install. `__init__.py` still reports `__version__ = "1.0.0"` while `pyproject.toml` is at `1.1.0`. `primary_key_strategy` is accepted, documented in the docstring, and never read — the generators always emit a surrogate key. `numpy` is imported in `type_detector.py` and unused. None of these break anything, and all of them are the kind of thing a reviewer catches in ten seconds; shipping solo with no second reader is precisely how they survived to release. The other thing I would change is the v1.1.0 commit. It rewrote the README around cost-savings tables — dollar figures per project, monthly BigQuery savings — built entirely from published industry rates and pricing pages, not from anything this code was measured doing. The caveat line is there, but the tables read as results. I would rather have spent that commit writing a benchmark that produces one honest number than writing six persuasive ones.

### Outcome

Shipped as an installable Python package. Two releases are built in `dist/`: `schema_genie-1.0.0` (7 Mar 2026, the full package — 28 files, ~2,000 lines) and `schema_genie-1.1.0` (8 Mar 2026, documentation plus version bump), each as a wheel and an sdist. What ships: a six-stage inference pipeline over pandas DataFrames, DDL generation for Snowflake, Redshift, BigQuery and PostgreSQL, SCD Type 2 audit columns on every generated dimension table, YAML-driven configuration via `SchemaGenie.from_config`, direct deployment to a live warehouse connection, and optional Graphviz ER-diagram export. 68 tests collected and passing under pytest, concentrated on type detection, fact selection and the four generators.

### Overview

schema-genie eliminates 80–90% of manual data-modelling work. Given one or more pandas DataFrames, its 6-stage inference pipeline classifies column semantic types, detects FK relationships, scores and selects fact tables, recommends star vs snowflake topology, detects SCD Type 2 candidates, and emits fully-annotated, warehouse-specific DDL — complete with surrogate keys, audit columns, distribution keys, indexes, and FK constraints. Supports live deployment, ER diagram export, and YAML configuration.

### Features

- 6-stage inference pipeline: type detection through DDL generation
- Semantic column type classification (id, date, currency, measure, category, text)
- FK relationship detection via value-set intersection scoring
- Fact table selection with composite scoring formula
- Star vs snowflake schema recommendation
- SCD Type 2 candidate detection with audit columns
- DDL for Snowflake, Redshift, BigQuery, and PostgreSQL
- ER diagram export (graphviz PNG/SVG/PDF)

### Stack

Python, Pandas, NumPy, sqlglot, PyYAML, Graphviz, Snowflake, BigQuery, Redshift, PostgreSQL

## Pattern Drift

URL: https://muhammadsufiyanbaig.vercel.app/projects/package/pattern-drift
Category: package
Timeline: Mar 2026 – Jun 2026
Role: Solo. I own every line of the library — the five-stage pipeline in `pattern_drift/`, all four detector implementations (ADWIN, Page-Hinkley, KSWIN, DDM) written from the source papers against the Python standard library only, the 14-file / 2,229-line test suite against 1,317 lines of library code, the packaging config, and the reference docs (`docs/system-report.md`, `docs/configuration-report.md`). There was no second reviewer, which shows up later in this document.
Links: Code: https://github.com/muhammadsufiyanbaig/pattern-drift | PyPI: https://pypi.org/project/pattern-drift/

Streaming-native concept drift detection library with four canonical algorithms, drift type classification, retraining window recommendation, and multi-channel alerting — zero mandatory dependencies.

### Problem

Deployed models decay quietly. The data feeding them shifts — a new customer segment, a pricing change, a sensor that starts reading half a degree high — and nothing in the pipeline reports it, because every stage still returns a number. Teams cope in two bad ways: retrain on a blind calendar schedule and pay for the runs whether or not anything changed, or wait for a business metric to sag and work backward, weeks late. Worse, once someone finally accepts the model has gone stale, nobody can say *which slice of history is still clean enough to retrain on* — so the usual answer is "the last 90 days", chosen because it is round.

### Constraints

The library had to install into pipelines that already existed, without asking anyone to adopt a monitoring platform or add a dependency their security review would have to clear — which ruled out scipy for the statistics and pandas for the data handling. It had to work on a stream, one record at a time, with bounded memory, because the target is an inference handler or a Kafka consumer loop, not a batch job with the whole dataset in hand. The four algorithms come from papers spanning 1954 to 2020, several of them paywalled: of the 22 references in the system report, 12 had a downloadable open-access PDF and 3 were dropped entirely because no version was locatable (`docs/papers/DOWNLOAD_LOG.md`). Page-Hinkley and DDM were implemented from secondary descriptions rather than the originals.

### Architecture

Five stages, one per file, because the hard part of drift detection is not the statistics — it is
that four different algorithms disagree about what a "drift signal" even is, and the pipeline
around them has to not care. `BaseDetector` is deliberately two methods: `update(value) ->
(bool, float)` and `reset()`. Everything above it — classification, window selection, alerting —
consumes only that tuple, so adding a detector means writing a class and adding one line to
`_METHODS` in `monitor.py`. The cost of that narrowness is real and visible in the code: DDM
consumes prediction-correctness signals rather than feature values, and the interface has nowhere
to say so, so `sensitivity` is accepted and silently ignored for DDM
(`docs/configuration-report.md` §3.4) and Page-Hinkley's `lambda_` stays pinned at 50.0 with no way
to reach it through `DriftMonitor`.

The bigger decision was one detector *per feature* instead of one multivariate test over the whole
row. Per-feature means the alert names the column that moved, which is what makes it actionable,
and it means a stable feature can never be dragged into an alarm by a noisy neighbour — there is a
whole test file enforcing exactly that (`tests/test_multi_feature_isolation.py`). The tradeoff is
that correlational drift is invisible: if two features shift together such that each marginal
stays put, nothing fires. That is documented as a limitation, not solved. Stage 4 is the piece
that does not exist in the comparable libraries — when drift fires, `RetrainingWindowEngine` walks
the score history backward to the last continuous run below the sensitivity threshold, trims 10%
off each end to avoid edge contamination, and hands back concrete indices instead of leaving the
"retrain on what?" question open.

```flow
Client :: Streaming producer calls DriftMonitor.update(record) — dict, pandas Series, or DataFrame micro-batch
FeatureExtractor :: Coerces to {feature: float}, drops non-numeric columns, auto-discovers feature names on first call
Detector pool :: One BaseDetector per feature (ADWIN | PageHinkley | KSWIN | DDM), each returns (drift_detected, score)
DriftClassifier :: 50-observation deque per feature; labels the event sudden / recurring / incremental / gradual by signal shape
RetrainingWindowEngine :: Walks _score_history backward to the last stable run, trims 10% per end, returns start/end/n_samples/confidence
AlertDispatcher :: Fires registered callbacks (Slack, webhook, log, custom), each wrapped so user code cannot kill the stream
DriftResult :: Returned to caller; _score_history kept in memory bounded by max_window, exportable to JSON or CSV
```

### Challenges

- The KS test without scipy: KSWIN needs a two-sample Kolmogorov-Smirnov p-value, and scipy is a ~30 MB dependency I had refused to take. Computing the D statistic is a merge-walk over two sorted arrays; the p-value is the hard half, because the Kolmogorov distribution is an infinite alternating series that oscillates badly near zero. What worked was the Kolmogorov–Smirnov asymptotic approximation with the `(√n + 0.12 + 0.11/√n)` finite-sample correction, the series truncated at 100 terms, and hard cutoffs at `z < 0.2 → 1.0` and `z > 5.0 → 0.0` where the series stops being numerically worth evaluating (`kswin.py:124-138`).
- Detector scores were on four incompatible scales: ADWIN produces a ratio against a Hoeffding bound, Page-Hinkley a cumulative sum against λ, KSWIN a p-value, DDM a deviation from a running minimum. Everything downstream — the classifier's `< 0.2` / `> 0.6` shape thresholds, the window engine's comparison against `sensitivity`, the timeline plot — assumes one shared scale. I normalised all four into roughly `[0, 1]`, but ADWIN and Page-Hinkley get there through divisors (`min(score / 10.0, 1.0)`, `U / lambda_`) that are calibration constants with no derivation behind them. It works because the tests pin behaviour rather than values; it is the part of the codebase I trust least.
- The retraining window's confidence score is structurally meaningless: `find_window()` computes confidence as the fraction of samples in the stable region that are below the sensitivity threshold — but the region was selected by walking backward *until* a sample exceeded that threshold, so every sample in it passes by construction. Confidence is always exactly 1.0. I verified this against a synthetic history with an unstable spike and got `RetrainingWindowResult(start=65, end=184, n_samples=120, confidence=1.0)`. The tests missed it because they assert `confidence >= 0.8` and `0.0 <= confidence <= 1.0` — both true of a constant (`tests/test_retraining_window.py:64,85`). The field ships as a stub and the number should not be trusted until the calculation is redone over a window that can actually contain instability.
- Refusing to depend on sklearn broke sklearn integration: `DriftDetector` implements `fit`/`transform`/`get_params`/`set_params` by hand instead of inheriting `BaseEstimator`, so that `pip install pattern-drift` never pulls scikit-learn. That worked under the duck-typed estimator protocol and stopped working when sklearn moved to `__sklearn_tags__`: on sklearn 1.8.0, all three `TestPipelineIntegration` tests fail with `'DriftDetector' object has no attribute '__sklearn_tags__'`, while the 15 standalone wrapper tests still pass. The isolated wrapper is fine; the thing the wrapper exists for is broken. The fix is a lazy `BaseEstimator` import behind the optional extra, which trades the clean import graph for a live integration — not yet done.
- Packaging failed on details the code could not catch: The initial release shipped `build-backend = "setuptools.backends.legacy:build"`, which is not a real backend, alongside `license = { text = "MIT" }` and a `License :: OSI Approved` classifier that modern setuptools now rejects as a pair. Nothing in 169 tests touches `pyproject.toml`, so this surfaced only at build time and took its own release (`da6b6e9`, 0.1.0 → 0.1.1) to correct. `__init__.py` still reports `__version__ = "0.1.0"` — the bump never propagated past the manifest.

### New ground covered

- Implementing statistical tests from papers rather than calling them: I had used KS tests and CUSUM as library functions. Writing them meant reading Hoeffding's bound closely enough to place the harmonic mean of sub-window sizes correctly in `ε_cut = √((1/2m)·ln(4n/δ))`, and discovering that the direction of a "sensitivity" knob is not a shared convention — ADWIN's `δ` and KSWIN's `α` are significance levels where larger means *earlier* detection, Page-Hinkley's `δ` is a minimum detectable magnitude where smaller means earlier. Exposing them all as one `sensitivity` parameter means the parameter reverses meaning depending on `method`, which is why `tests/test_sensitivity_calibration.py` exists as its own file: to pin the monotonicity direction per algorithm before anyone relies on it.
- False-positive-rate testing as a first-class test category: I was used to testing that code detects the thing. Here the more useful test is that it *doesn't* — 1,000 samples of stable Gaussian noise through each detector, asserting the alarm rate stays under its theoretical bound (`tests/test_false_positive_rate.py`). A drift detector that fires on stable data is worse than no detector, because it trains the on-call engineer to ignore it. This is the category that caught the most broken intermediate implementations.
- Zero-mandatory-dependency packaging with optional extras: Five extras (`viz`, `alerts`, `yaml`, `pandas`, `sklearn`), each guarded by a `try: import` at the call site that raises an ImportError naming the exact install command. What I understand now that I did not: this pattern makes the dependency graph clean and the *test* graph a liability, because the code paths behind each extra only run when someone happens to have the package installed — the sklearn breakage had every ingredient to sit undetected in a CI environment that pinned an older sklearn.

### Takeaways

- A test that asserts a bound rather than a value will pass forever against a constant, which is how a hardcoded `confidence = 1.0` survived twelve tests written specifically to check it.
- The interesting output of a drift detector is not the alarm but the answer to "retrain on what?" — and that answer is the part no comparable library ships.
- Refusing a dependency is a real, defensible decision that transfers the maintenance burden onto you: sklearn's estimator protocol changed, and because I had opted out of inheriting from it, the break was mine to absorb.
- Four algorithms behind one `sensitivity` parameter is a friendly API that quietly lies, because the parameter's direction inverts between ADWIN and Page-Hinkley — the abstraction needs either per-algorithm names or per-algorithm documentation, and I chose documentation.
- Packaging metadata is untested code with a release-shaped blast radius; 169 passing tests said nothing about whether the wheel would build.

### Journey

I started this because of the retraining-schedule problem, not the detection problem. Detection had four decent published answers; what nobody shipped was the thing that comes after the alarm. That framing is why `window_engine.py` exists at all, and it is the reason the pipeline has five stages rather than stopping at three. I got the dependency question backwards at first. My instinct was to reach for scipy for the KS test and pandas for input handling, and I spent a while there before accepting that a monitoring library asking for a 30 MB scientific stack is a library that gets declined in review. Rewriting the KS p-value against `math` alone was the moment the design clicked — once I was forced to hold the whole statistical core in the standard library, the detector interface collapsed to two methods and everything above it got simpler. The optional-extras pattern fell out of that constraint rather than being planned. The thing I got most wrong was trusting my own test suite. I wrote 169 tests across 14 categories and told myself the surface was covered. Then I actually ran everything on a current environment for this write-up and found three sklearn integration tests failing on an API change, plus a confidence score that is arithmetically incapable of returning anything but 1.0 despite a dedicated test file. Both had been sitting there. Neither is exotic — one is a dependency moving, one is a loop invariant I did not think through — and both are exactly what a second reader or a CI job would have caught in an afternoon. What I would do differently: put CI in before the first release, not after — this repo has no `.github/` and no workflow file, and a matrix across sklearn versions would have caught the `__sklearn_tags__` break the week it landed. And I would stop writing the report before the implementation is final. `docs/system-report.md` claims 175 tests with 151 passing; the actual suite collects 169 and reports 166 passed, 3 failed, 1 skipped. The document drifted from the code it describes, which is a slightly humiliating thing to have happen in a project about drift.

### Outcome

Shipped as `pattern-drift` 0.1.1 on 18 Jun 2026 — wheel and sdist built into `dist/`, Python 3.9+, zero mandatory runtime dependencies, MIT. Four detection algorithms behind one interface, per feature isolation, four-way drift-type classification, a retraining-window recommendation, and a callback dispatcher with Slack/webhook/logging built in. Optional extras cover matplotlib, requests, pyyaml, pandas, and scikit-learn. Verified state as of this review, on Python 3.13.14 / sklearn 1.8.0 / pandas 2.3.3: **169 tests collected, 166 passed, 3 failed, 1 skipped.** The three failures are all `tests/test_sklearn_pipeline.py::TestPipelineIntegration`, blocked on `__sklearn_tags__`; the skip is the Hypothesis property-based module, which self-skips when `hypothesis` is not installed. Two known defects are open: the always-1.0 retraining-window confidence, and `__init__.py` reporting `__version__ = "0.1.0"` against a 0.1.1 manifest. The library is documented against 12 downloaded source papers with 3 citations dropped for lack of open access (`docs/papers/DOWNLOAD_LOG.md`). 1.1 was published to PyPI or only built locally — `dist/` has the artifacts, the repo has no publish step. The cost model in `docs/system-report.md` §10.2 is a set of assumed inputs, not measurements, and should not be quoted as a result.

### Overview

pattern-drift is a pure-Python, zero-dependency library for continuous, per-feature drift detection in production ML pipelines. It integrates ADWIN, Page-Hinkley, KSWIN, and DDM detectors in a five-stage pipeline: feature extraction → per-feature detection → temporal drift classification (sudden, gradual, incremental, recurring) → retraining window recommendation → alert dispatching. Includes a scikit-learn Pipeline wrapper, YAML config support, JSON/CSV report export, and 175 passing tests.

### Features

- Four detection algorithms: ADWIN, Page-Hinkley, KSWIN, DDM
- Per-feature independent detector instances
- Drift type classification: sudden, gradual, incremental, recurring
- Retraining window recommendation with confidence scoring
- scikit-learn Pipeline transformer integration
- Zero mandatory runtime dependencies
- Built-in Slack, HTTP webhook, and logging callbacks
- 175 tests across 14 test files — 151 passing

### Stack

Python, Pandas, scikit-learn, Matplotlib, PyYAML, Hypothesis, pytest

## Simple VCS

URL: https://muhammadsufiyanbaig.vercel.app/projects/package/simple-vcs
Category: package
Timeline: Jun 2025 – Feb 2026 (three bursts of work, not continuous)
Role: Solo. I own everything in the repo: the storage format (`.svcs/` with a content-addressed `objects/` directory plus three JSON files), the `SimpleVCS` class in [core.py](simple_vcs/core.py) that implements all ten operations, the Click CLI in [cli.py](simple_vcs/cli.py) that wraps them, the packaging (`setup.py` + `pyproject.toml`, console script `svcs`), and the GitHub Actions workflows. There is no second contributor — `git log` shows one author under two email addresses (a university address for the 2025 commits, a personal one afterwards).
Links: Live: https://pypi.org/project/simple-vcs/ | Code: https://github.com/muhammadsufiyanbaig/simple_vcs

A lightweight version control system built from scratch in Python — published on PyPI with CLI and Python API interfaces.

### Problem

Git is the correct answer for source code and the wrong answer for a person who just wants their file history back. The commands that matter to a beginner — "save this", "show me what I saved", "put it back the way it was" — are buried under staging semantics, detached HEADs, and a reflog you have to know exists before it can save you. People coping without it fall back to `report_final_v2_actually_final.docx` and a folder of dated ZIPs, which works right up until they need to know which copy was current on which day. The second problem was mine: I wanted to actually understand how a version control system stores things, and reading about content-addressed objects is not the same as writing one and finding out where it leaks.

### Constraints

It had to be installable by someone who only has `pip`, so no database, no daemon, no compiled dependency — the entire store is plain files and JSON that a human can open in a text editor when something goes wrong. It had to run identically on Windows, macOS and Linux (the root [workflow.yml](workflow.yml) matrix covers all three), and Windows console encoding turned out to be the single most persistent source of breakage. `pyproject.toml` declares `requires-python = ">=3.7"`, which rules out newer typing and pathlib conveniences. And it is a side project — the gaps between the three work bursts are five months and one month, so every design had to survive me forgetting all of it and coming back cold.

### Architecture

Two layers, deliberately thin. [cli.py](simple_vcs/cli.py) is Click and nothing else: each
command constructs a `SimpleVCS` and calls one method. All state lives in
[core.py](simple_vcs/core.py). The split exists so the same class is usable as a Python API —
`from simple_vcs import SimpleVCS` — and the root workflow tests that path separately from the
CLI path, asserting on the boolean return of every method. That is why every public method
returns `bool` instead of raising: the return value *is* the API contract, and the Rich output
is a side effect on top of it.

The store is content-addressed but only halfway. `_calculate_file_hash` reads in 4 KB chunks
and produces a SHA-256; `_store_object` writes the bytes to `.svcs/objects/<hash>` and skips
the write if that hash already exists, so re-committing an unchanged file costs nothing. But
commits themselves are not objects — they are appended to a single `commits.json` array with a
monotonic integer `id`, and `HEAD` is a text file containing that integer. I chose integers
over commit hashes because `svcs revert 3` is something a person can type from memory and
`svcs revert a5f62c6` is not. The cost is real and I knew it going in: a JSON array means the
whole history is rewritten on every commit, and an integer counter derived from
`len(commits) + 1` means there is no safe way to grow branches. Diffing inherits the same
tradeoff — `show_diff` compares the *file-hash sets* of two commits, so it reports added,
deleted and modified plus a byte-size delta, but never line content. Storing whole blobs
instead of deltas made that a design consequence, not an oversight.

```flow
svcs CLI (Click) :: Parses argv, one command per SimpleVCS method, RichGroup overrides help rendering
SimpleVCS (core.py) :: Validates repo exists, resolves paths, enforces the file-inside-repo rule
_calculate_file_hash :: SHA-256 over 4KB chunks, gives the content address
.svcs/objects/<sha256> :: Raw file bytes written once per unique hash, skipped if already present
.svcs/staging.json :: Path to {hash, size, mtime} for what is staged but not committed
.svcs/commits.json :: Append-only array; each commit snapshots the whole staging map plus a parent id
.svcs/HEAD :: Plain-text integer commit id, "0" means no commits yet
Rich console :: Panels and tables rendered from the same data, forced terminal mode for Windows
```

### Challenges

- The CLI entry point was dead for eight months: Commit `e62ec27` changed `if __name__ == '__main__': main()` to `main` — dropping the call parentheses. Running the module did nothing at all and exited zero, which is the worst possible failure because nothing complains. It survived June 2025 to January 2026 and was only fixed in `144c16b` ("fixed CLI entry point"). The reason it hid so long is that CI never invoked that path — the workflows call `python -m simple_vcs.cli --help`, which goes through Click's console-script entry, not through `__main__`.
- `add` accepted files outside the repository: The first version did `Path(file_path)` without resolving, then called `.relative_to(self.repo_path)` on it later. A relative path from a different working directory would either resolve wrong or throw a raw `ValueError` at the user. The fix in `6a7cd84` was to `.resolve()` immediately and wrap `relative_to` in a try/except that turns the containment check into an explicit error message. It is now the only place in the codebase where an exception is used as a boundary test on purpose.
- Windows console encoding: `SimpleVCS.__init__` constructs `Console(force_terminal=True, legacy_windows=False)` and stores `self.is_windows`, and the comment above it says "Force UTF-8 output and disable emoji on Windows". Every emoji in the v1.2 output was replaced with ASCII markers — the current commit marker in `show_log` is `->`, not an arrow glyph, and the diff table uses `+ Added` / `- Deleted` / `M Modified`. The CHANGELOG for v1.2 still documents asterisks and colored circles that the shipped code no longer emits; the docs lost that fight and I did not go back to fix them.
- flake8 choked on a UTF-16 file I had committed: `run_svcs.py` was written by a PowerShell redirect, so it is UTF-16 with a BOM — every character is followed by a null byte. flake8 reports that as a source error, and because CI runs `flake8 . --select=E9,F63,F7,F82` as a hard gate, the build failed on a one-line convenience script. The last commit (`3ec145c`) added a [.flake8](.flake8) with an exclude list covering `build`, `dist`, `*.egg-info`, `test_project` and `.github`, and deleted the checked-in `simple_vcs.egg-info/`. That fixed the build. It did not fix `run_svcs.py`, which is still UTF-16 in the tree.
- `compress` does not compress anything: `compress_objects` gzips each object over 1 KB to a `.gz` file, deletes the original, then immediately decompresses the `.gz` back to the original filename and deletes the `.gz` — "Decompress back to original name for compatibility". The net effect on disk is zero bytes saved, and the panel that reports "Space saved" is always reporting 0. I wrote the round-trip because nothing in the read path knows how to open a gzipped object, so leaving them compressed would have broken `revert`. The honest fix is a format flag in the object header and a decompressing read path; that work is not done, and the command should not have shipped in this state.

### New ground covered

- Content-addressed storage: I had used Git for years without knowing what was under it. Writing `_store_object` is what made it concrete: the `if not obj_path.exists()` guard is the entire deduplication story, and identity-by-content is what makes it safe. What I understand now that I did not before is why Git separates blobs from commits — I collapsed them into one JSON array and immediately lost the ability to branch.
- Rich: v1.2 was a full rewrite of the output layer from `print()` to Rich panels, tables and trees — the diff touched 403 lines of `core.py` in one commit. The lesson was not the API; it was that Rich pushed presentation logic into `core.py`, where it does not belong. Every method now formats its own byte sizes inline, duplicating `format_file_size` in [utils.py](simple_vcs/utils.py), which is now dead code.
- Click group subclassing: `RichGroup` overrides `format_help` to render my own help screen, then sets `ctx.resilient_parsing = True` to stop Click printing its version underneath. That flag is not what it is documented for — I am using a parsing hint as an output suppressor because Click gives no cleaner hook. It works and I am not proud of it.
- PyPI packaging and trusted publishing: The root [workflow.yml](workflow.yml) builds with `python -m build`, validates with `twine check`, publishes to TestPyPI on tags and to real PyPI only on a published release. Learning that the TestPyPI dry run is the step that catches broken metadata — before it is permanently on the real index — was worth the setup.

### Takeaways

- A CLI bug that makes the program exit successfully while doing nothing is far more dangerous than one that crashes, and CI that only tests the packaged entry point will never see it.
- Integer commit IDs bought real usability at the cost of branching; that tradeoff was correct for this project's audience and would be wrong for almost any other version control system.
- Committing build artifacts — `dist/`, `build/`, `*.egg-info/` — cost me a broken lint gate and three separate cleanup commits before I added a `.gitignore` at all.
- Presentation code that reaches into the core spreads: once `core.py` owned the Rich console, ten methods each grew their own byte-size formatter and the shared utility died.
- A feature whose output claims a result it does not produce is worse than a missing feature, because the user has no reason to doubt the number on the screen.

### Journey

I started this because I wanted to know what Git actually does when you type `git add`, and the fastest way to find out is to try to build the smallest thing that behaves the same. The first day produced most of what still exists — init, add, commit, log, diff, status — under plain `print()`, and it worked. Then it sat for five months. What I got wrong early is visible in the git history and it is a little embarrassing. Six of my first seven commits are named "changes" or "complete", `dist/` and `__pycache__/` were committed for months, `setup.py` pointed at `github.com/yourusername/simple-vcs` for four days, and `__init__.py` credited "Your Name". Those are the marks of writing a package by filling in a template without reading it. The single worst one is the dropped parentheses in `e62ec27` — one character, in a commit called "changes", that made `python run_svcs.py` a no-op for eight months and that I did not find until I came back in January 2026 to add features. The moment it clicked was `_store_object`. Three lines: hash the content, write it under the hash, skip if it is already there. Deduplication, integrity and immutability all fall out of one idea, and I stopped thinking of Git's object store as machinery and started thinking of it as a consequence. Everything I like about this codebase is downstream of that afternoon. The v1.2 rewrite was the other kind of learning. I spent a full session converting every `print()` to a Rich panel and it genuinely made the tool feel finished — but I put all of it in `core.py`, so the module that owns the storage format now also owns the border style of the commit table. Two versions later that decision has metastasized: `utils.format_file_size` is dead because ten methods each inline their own copy. What I would do differently: write the tests. There is not a single test file in this repository, and the CI job that says "Test with pytest" runs bare `pytest` against zero tests — which pytest treats as an error, so that job has been failing rather than passing vacuously. The real testing lives in [workflow.yml](workflow.yml), which drives the CLI through a full init/add/commit/log/diff sequence in a temp directory and asserts on the Python API's return values. That file is a decent integration suite and it is sitting in the repo root instead of `.github/workflows/`, so GitHub has never run it. Moving it and writing real unit tests around `_calculate_file_hash`, `quick_revert` and the containment check in `add_file` is the first thing I would do on picking this back up — closely followed by either fixing `compress` or removing it.

### Outcome

Shipped as `simple-vcs` version 1.3.0 (Feb 2026), built into `dist/simple_vcs-1.3.0-py3-none-any.whl` and `.tar.gz`, installing a `svcs` console script. Ten commands: `init`, `add`, `commit`, `status`, `log`, `diff`, `revert`, `snapshot`, `restore`, `compress`. Both a CLI and an importable Python API, verified separately by the workflow in [workflow.yml](workflow.yml) across Python 3.7–3.12 on Ubuntu, Windows and macOS. Version numbers are currently inconsistent across three files and that is a shipping defect, not a rounding error: `pyproject.toml` and `setup.py` say `1.3.0`, `click.version_option` in [cli.py](simple_vcs/cli.py) says `1.3.0`, the help footer four lines below it still prints `Version 1.2.0`, and `simple_vcs/__init__.__version__` is still `1.1.0`. yml](workflow.yml) only fires on a GitHub release event, but there are no tags in this repo. github/workflows/python-package.yml` pytest step is currently red on main; the repo has no test files, so it should be, but the run history is not in the tree.

### Overview

Simple VCS is a fully functional version control system inspired by Git, built entirely from scratch in Python. It lets developers track file changes, commit history, and restore previous states without Git overhead. Published on PyPI as `simple-vcs` (v1.3.0), it supports the core VCS workflow including init, add, commit, log, diff, and status commands. It features quick revert to any previous commit by ID, zip-based full project backups, and a rich terminal UI using the `rich` library. Cross-platform compatible with Windows and Unix.

### Features

- Core VCS workflow — init, add, commit, log, diff, status
- Quick revert — instantly restore any previous commit by ID
- Snapshot & restore — zip-based full project backups
- Object compression — automatic storage optimization
- Dual interface — CLI (svcs command) + importable Python API
- Rich terminal UI — styled tables, panels, and colored output
- Cross-platform — Windows & Unix compatible

### Stack

Python, click, rich, SHA-256 hashing, JSON-based storage, zipfile

---

# Skills

Levels are self-assessed against production usage, not tutorials.

## language

- JavaScript — expert
- TypeScript — expert
- Python — expert

## frontend

- React.js — expert
- Next.js — expert
- Tailwind CSS — expert
- Material UI — solid
- ShadCn UI — advanced
- Redux — advanced
- Zustand — advanced
- Framer Motion — advanced

## backend

- Node.js — expert
- Express.js — advanced
- FastAPI — expert
- GraphQL — advanced
- Kafka — solid

## database

- MySQL — advanced
- Neon Postgres — expert
- SQLite — solid
- Firebase — solid
- Redis — advanced
- Drizzle ORM — expert

## ai

- Groq — solid
- LangChain — advanced

## tools

- Sanity.io — solid

## devops

- Vercel — advanced
- Docker — advanced
- Git — advanced

---

## Contact

- Portfolio: https://muhammadsufiyanbaig.vercel.app
- Email: send.sufiyan@gmail.com
- Resume: https://muhammadsufiyanbaig.vercel.app/MuhammadSufiyanBaig.pdf
- github: https://github.com/muhammadsufiyanbaig
- linkedin: https://www.linkedin.com/in/muhammadsufiyanbaig/
- twitter: https://x.com/Sufiyan395
- email: mailto:send.sufiyan@gmail.com