From 59d53967b0cba71da9a4db2168a17a05deb09806 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Luk=C3=A1=C5=A1=20Trkan?= Date: Thu, 13 Nov 2025 01:35:13 +0100 Subject: [PATCH] update report --- 7project/report.md | 102 +++++++++++++++++++++++++++++---------------- 1 file changed, 66 insertions(+), 36 deletions(-) diff --git a/7project/report.md b/7project/report.md index 70bff9c..f3dcd93 100644 --- a/7project/report.md +++ b/7project/report.md @@ -22,7 +22,7 @@ filtering and visualization. New transactions are automatically fetched in the b ## Architecture Overview Our system is a full‑stack web application composed of a React frontend, a FastAPI backend, -a PostgreSQL database, and asynchronous background workers powered by Celery with RabbitMQ. +a MariaDB database with Maxscale, and asynchronous background workers powered by Celery with RabbitMQ. Redis is available for caching/kv and may be used by Celery as a result backend. The backend exposes REST endpoints for authentication (email/password and OAuth), users, categories, transactions, exchange rates and bank APIs. A thin controller layer (FastAPI routers) lives under app/api. @@ -33,15 +33,17 @@ the application is packaged via a Helm chart. ```mermaid flowchart LR - proc_queue[Message Queue] --> proc_queue_worker[Worker Service] - proc_queue_worker --> ext_mail[(Email Service)] - proc_cron[Cron] --> svc - proc_queue_worker --> ext_bank[(Bank API)] - proc_queue_worker --> db - client[Client/Frontend] <--> svc[Backend API] + n3(("User")) <--> client["Frontend"] + proc_queue["Message Queue"] --> proc_queue_worker["Worker Service"] + proc_queue_worker -- SMTP --> ext_mail[("Email Service")] + proc_queue_worker <-- HTTP request/response --> ext_bank[("Bank API")] + proc_queue_worker <--> db[("Database")] + proc_cron["Cron"] <-- HTTP request/response --> svc["Backend API"] svc --> proc_queue - svc <--> db[(Database)] - svc <--> api[(UniRate API)] + n2["Cloudflare tunnel"] <-- HTTP request/response --> svc + svc <--> db + svc <-- HTTP request/response --> api[("UniRate API")] + client <-- HTTP request/response --> n2 ``` The workflow works in the following way: @@ -350,6 +352,7 @@ You can now use k8s client like https://headlamp.dev/ with the generated kubecon ```bash cp terraform.tfvars.example terraform.tfvars ``` + - `metallb_ip_range` - set to range available in your network for load balancer services - `mariadb_password` - password for internal mariadb user - `mariadb_root_password` - password for root user @@ -384,8 +387,10 @@ tofu apply ``` ### Configure deployment + 1) Create self-hosted runner with access to the cluster or make cluster publicly accessible -2) Change `jobs.deploy.runs-on` in `.github/workflows/deploy-prod.yml` and in `.github/workflows/deploy-pr.yaml` to your runner label +2) Change `jobs.deploy.runs-on` in `.github/workflows/deploy-prod.yml` and in `.github/workflows/deploy-pr.yaml` to your + runner label 3) Add variables to GitHub in repository settings: - `PROD_DOMAIN` - base domain for deployments (e.g. ltrk.cz) - `DEV_FRONTEND_BASE_DOMAIN` - base domain for your cloudflare pages @@ -413,8 +418,8 @@ tofu apply - UNIRATE_API_KEY 5) On Github open Actions tab, select "Deploy Prod" and run workflow manually - # TODO: REMOVE I guess + 2) Deploy the app using Helm ```bash @@ -432,10 +437,6 @@ helm upgrade --install myapp charts/myapp-chart \ --set env.SECRET="CHANGE_ME_SECRET" ``` - - - - ## Testing Instructions The tests are located in 7project/backend/tests directory. All tests are run by GitHub actions on every pull request and @@ -554,18 +555,18 @@ curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/authenticated-route > This information is used for individual grading. > Link to the specific commit on GitHub for each contribution. -| Task/Component | Assigned To | Status | Time Spent | Difficulty | Notes | -|-------------------------------------------------------------------------------------------------------------------|-------------|----------------|------------|------------|-------------| -| [Project Setup & Repository](https://github.com/dat515-2025/Group-8#) | Lukas | βœ… Complete | [X hours] | Medium | [Any notes] | -| [Design Document](https://github.com/dat515-2025/Group-8/blob/main/6design/design.md) | Both | βœ… Complete | 4 Hours | Easy | [Any notes] | -| [Backend API Development](https://github.com/dat515-2025/Group-8/tree/main/7project/backend/app/api) | Dejan | βœ… Complete | 12 hours | Medium | [Any notes] | -| [Database Setup & Models](https://github.com/dat515-2025/Group-8/tree/main/7project/backend/app/models) | Lukas | πŸ”„ In Progress | [X hours] | Medium | [Any notes] | -| [Frontend Development](https://github.com/dat515-2025/Group-8/tree/main/7project/frontend) | Dejan | βœ… Complete | 17 hours | Medium | [Any notes] | -| [Docker Configuration](https://github.com/dat515-2025/Group-8/blob/main/7project/compose.yml) | Lukas | βœ… Complete | [X hours] | Easy | [Any notes] | -| [Cloud Deployment](https://github.com/dat515-2025/Group-8/blob/main/7project/deployment/app-demo-deployment.yaml) | Lukas | βœ… Complete | [X hours] | Hard | [Any notes] | -| [Testing Implementation](https://github.com/dat515-2025/group-name) | Dejan | βœ… Complete | 16 hours | Medium | [Any notes] | -| [Documentation](https://github.com/dat515-2025/group-name) | Both | πŸ”„ In Progress | [X hours] | Easy | [Any notes] | -| [Presentation Video](https://github.com/dat515-2025/group-name) | Both | ❌ Not Started | [X hours] | Medium | [Any notes] | +| Task/Component | Assigned To | Status | Time Spent | Difficulty | Notes | +|-------------------------------------------------------------------------------------------------------------------|-------------|----------------|------------|------------|-----------------------------------------------------------------------------------------------------| +| [Project Setup & Repository](https://github.com/dat515-2025/Group-8#) | Lukas | βœ… Complete | [X hours] | Medium | [Any notes] | +| [Design Document](https://github.com/dat515-2025/Group-8/blob/main/6design/design.md) | Both | βœ… Complete | 4 Hours | Easy | [Any notes] | +| [Backend API Development](https://github.com/dat515-2025/Group-8/tree/main/7project/backend/app/api) | Dejan | βœ… Complete | 12 hours | Medium | [Any notes] | +| [Database Setup & Models](https://github.com/dat515-2025/Group-8/tree/main/7project/backend/app/models) | Lukas | βœ… Complete | [X hours] | Medium | [Any notes] | +| [Frontend Development](https://github.com/dat515-2025/Group-8/tree/main/7project/frontend) | Dejan | βœ… Complete | 17 hours | Medium | [Any notes] | +| [Docker Configuration](https://github.com/dat515-2025/Group-8/blob/main/7project/compose.yml) | Lukas | βœ… Complete | 3 hours | Easy | [Any notes] | +| [Cloud Deployment](https://github.com/dat515-2025/Group-8/blob/main/7project/deployment/app-demo-deployment.yaml) | Lukas | βœ… Complete | [X hours] | Hard | Using Talos cluster running in proxmox - easy snapshots etc. Frontend deployed at Cloudflare pages. | +| [Testing Implementation](https://github.com/dat515-2025/group-name) | Dejan | βœ… Complete | 16 hours | Medium | [Any notes] | +| [Documentation](https://github.com/dat515-2025/group-name) | Both | πŸ”„ In Progress | [X hours] | Easy | [Any notes] | +| [Presentation Video](https://github.com/dat515-2025/group-name) | Both | ❌ Not Started | [X hours] | Medium | [Any notes] | **Legend**: βœ… Complete | πŸ”„ In Progress | ⏳ Pending | ❌ Not Started @@ -575,14 +576,27 @@ curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/authenticated-route ### [LukΓ‘Ε‘] -| Date | Activity | Hours | Description | -|----------------|---------------------|------------|----------------------------------------------------| -| 4.10 to 10.10 | Initial Setup | 40 | Repository setup, project structure, cluster setup | -| 14.10 to 16.10 | Backend Development | 12 | Implemented user authentication - oauth | -| 8.10 to 12.10 | CI/CD | 10 | Created database schema and models | -| [Date] | Testing | [X.X] | Unit tests for API endpoints | -| [Date] | Documentation | [X.X] | Updated README and design doc | -| **Total** | | **[XX.X]** | | +## Hour Sheet + +**Name:** LukΓ‘Ε‘ Trkan + +| Date | Activity | Hours | Description | Representative Commit / PR | +|:----------------|:----------------------------|:--------|:------------------------------------------------------------------------------------|:------------------------------------------------------| +| 18.9. - 19.9. | Initial Setup & Design | 40 | Repository init, system design diagrams, basic Terraform setup | `feat(infrastructure): add basic terraform resources` | +| 20.9. - 5.10. | Core Infrastructure & CI/CD | 12 | K8s setup (ArgoCD), CI/CD workflows, RabbitMQ, Redis, Celery workers, DB migrations | `PR #2`, `feat(infrastructure): add rabbitmq cluster` | +| 6.10. - 9.10. | Frontend Infra & DB | 5 | Deployed frontend to Cloudflare, setup metrics, created database models | `PR #16` (Cloudflare), `PR #19` (DB structure) | +| 10.10. - 11.10. | Backend | 5 | Implemented OAuth support (MojeID, BankID) | `feat(auth): add support for OAuth and MojeID` | +| 12.10. | Infrastructure | 2 | Added database backups | `feat(infrastructure): add backups` | +| 16.10. | Infrastructure | 4 | Implemented secrets management, fixed deployment/env variables | `PR #29` (Deployment envs) | +| 17.10. | Monitoring | 1 | Added Sentry logging | `feat(app): add sentry loging` | +| 21.10. - 22.10. | Backend | 8 | Added ČSAS bank connection | `PR #32` (Fix React OAuth) | +| 29.10. - 30.10. | Backend | 5 | Implemented transaction encryption, add bank scraping | `PR #39` (CSAS Scraping) | +| 30.10. | Monitoring | 6 | Implemented Loki logging and basic Prometheus metrics | `PR #42` (Prometheus metrics) | +| 9.11. | Monitoring | 2 | Added custom Prometheus metrics | `PR #46` (Prometheus custom metrics) | +| 11.11. | Tests | 1 | Investigated and fixed broken Pytest environment | `fix(tests): set pytest env` | +| 11.11. - 12.11. | Features & Deployment | 6 | Added cron support, email sender service, updated workers & image | `PR #49` (Email), `PR #50` (Update workers) | +| 18.9 - 14.11 | Documentation | 8 | Updated report.md, design docs, and tfvars.example | `Create design.md`, `update report` | +| **Total** | | **105** | | | ### Dejan @@ -610,16 +624,32 @@ curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/authenticated-route [Reflect on the key technical and collaboration skills learned during this project] ### Challenges Faced +#### Slow cluster performance +This was caused by single SATA SSD disk running all VMs. This was solved by adding second NVMe disk just for Talos VMs. + [Describe the main challenges and how you overcame them] ### If We Did This Again +#### Different framework +FastAPI lacks usable build in support for database migrations and implementing Alembic was a bit tricky. +Tricky was also integrating FastAPI auth system with React frontend, since there is no official project template. +Using .NET (which we considered initially) would probably solve these issues. + [What would you do differently? What worked well that you'd keep?] ### Individual Growth -#### [Team Member 1 Name] +#### [Lukas] +This course finally forced me to learn kubernetes (been on by TODO list for at least 3 years). +I had some prior experience with terraform/opentofu from work but this improved by understanding of it. + +The biggest challenge for me was time tracking since I am used to tracking to projects, not to tasks. +(I am bad even at that :) ). + +It was also interesting experience to be the one responsible for the initial project structure/design/setup +used not only by myself. [Personal reflection on growth, challenges, and learning]