lukas/uis-cloud-computing
1
0
Fork 0
mirror of https://github.com/dat515-2025/Group-8.git synced 2026-03-22 06:57:47 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

update report
Some checks are pending
Deploy Prod / Run Python Tests (push) Waiting to run
Details
Deploy Prod / Build and push image (reusable) (push) Blocked by required conditions
Details
Deploy Prod / Generate Production URLs (push) Blocked by required conditions
Details
Deploy Prod / Frontend - Build and Deploy to Cloudflare Pages (prod) (push) Blocked by required conditions
Details
Deploy Prod / Helm upgrade/install (prod) (push) Blocked by required conditions
Details

Browse Source
This commit is contained in:
Lukáš Trkan
2025-11-13 01:35:13 +01:00
parent f3086f8c73
commit 59d53967b0
1 changed files with 66 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
7project/report.md
View File

@@ -22,7 +22,7 @@ filtering and visualization. New transactions are automatically fetched in the b
## Architecture Overview ## Architecture Overview
Our system is a full‑stack web application composed of a React frontend, a FastAPI backend, Our system is a full‑stack web application composed of a React frontend, a FastAPI backend,
a PostgreSQL database, and asynchronous background workers powered by Celery with RabbitMQ. a MariaDB database with Maxscale, and asynchronous background workers powered by Celery with RabbitMQ.
Redis is available for caching/kv and may be used by Celery as a result backend. The backend Redis is available for caching/kv and may be used by Celery as a result backend. The backend
exposes REST endpoints for authentication (email/password and OAuth), users, categories, exposes REST endpoints for authentication (email/password and OAuth), users, categories,
transactions, exchange rates and bank APIs. A thin controller layer (FastAPI routers) lives under app/api. transactions, exchange rates and bank APIs. A thin controller layer (FastAPI routers) lives under app/api.
@@ -33,15 +33,17 @@ the application is packaged via a Helm chart.
```mermaid ```mermaid
flowchart LR flowchart LR
proc_queue[Message Queue] --> proc_queue_worker[Worker Service] n3(("User")) <--> client["Frontend"]
proc_queue_worker --> ext_mail[(Email Service)] proc_queue["Message Queue"] --> proc_queue_worker["Worker Service"]
proc_cron[Cron] --> svc proc_queue_worker -- SMTP --> ext_mail[("Email Service")]
proc_queue_worker --> ext_bank[(Bank API)] proc_queue_worker <-- HTTP request/response --> ext_bank[("Bank API")]
proc_queue_worker --> db proc_queue_worker <--> db[("Database")]
client[Client/Frontend] <--> svc[Backend API] proc_cron["Cron"] <-- HTTP request/response --> svc["Backend API"]
svc --> proc_queue svc --> proc_queue
svc <--> db[(Database)] n2["Cloudflare tunnel"] <-- HTTP request/response --> svc
svc <--> api[(UniRate API)] svc <--> db
svc <-- HTTP request/response --> api[("UniRate API")]
client <-- HTTP request/response --> n2
``` ```
The workflow works in the following way: The workflow works in the following way:
@@ -350,6 +352,7 @@ You can now use k8s client like https://headlamp.dev/ with the generated kubecon
```bash ```bash
cp terraform.tfvars.example terraform.tfvars cp terraform.tfvars.example terraform.tfvars
``` ```
- `metallb_ip_range` - set to range available in your network for load balancer services - `metallb_ip_range` - set to range available in your network for load balancer services
- `mariadb_password` - password for internal mariadb user - `mariadb_password` - password for internal mariadb user
- `mariadb_root_password` - password for root user - `mariadb_root_password` - password for root user
@@ -384,8 +387,10 @@ tofu apply
``` ```
### Configure deployment ### Configure deployment
1) Create self-hosted runner with access to the cluster or make cluster publicly accessible 1) Create self-hosted runner with access to the cluster or make cluster publicly accessible
2) Change `jobs.deploy.runs-on` in `.github/workflows/deploy-prod.yml` and in `.github/workflows/deploy-pr.yaml` to your runner label 2) Change `jobs.deploy.runs-on` in `.github/workflows/deploy-prod.yml` and in `.github/workflows/deploy-pr.yaml` to your
runner label
3) Add variables to GitHub in repository settings: 3) Add variables to GitHub in repository settings:
- `PROD_DOMAIN` - base domain for deployments (e.g. ltrk.cz) - `PROD_DOMAIN` - base domain for deployments (e.g. ltrk.cz)
- `DEV_FRONTEND_BASE_DOMAIN` - base domain for your cloudflare pages - `DEV_FRONTEND_BASE_DOMAIN` - base domain for your cloudflare pages
@@ -413,8 +418,8 @@ tofu apply
- UNIRATE_API_KEY - UNIRATE_API_KEY
5) On Github open Actions tab, select "Deploy Prod" and run workflow manually 5) On Github open Actions tab, select "Deploy Prod" and run workflow manually
# TODO: REMOVE I guess # TODO: REMOVE I guess
2) Deploy the app using Helm 2) Deploy the app using Helm
```bash ```bash
@@ -432,10 +437,6 @@ helm upgrade --install myapp charts/myapp-chart \
--set env.SECRET="CHANGE_ME_SECRET" --set env.SECRET="CHANGE_ME_SECRET"
``` ```
## Testing Instructions ## Testing Instructions
The tests are located in 7project/backend/tests directory. All tests are run by GitHub actions on every pull request and The tests are located in 7project/backend/tests directory. All tests are run by GitHub actions on every pull request and
@@ -555,14 +556,14 @@ curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/authenticated-route
> Link to the specific commit on GitHub for each contribution. > Link to the specific commit on GitHub for each contribution.
| Task/Component | Assigned To | Status | Time Spent | Difficulty | Notes | | Task/Component | Assigned To | Status | Time Spent | Difficulty | Notes |
|-------------------------------------------------------------------------------------------------------------------|-------------|----------------|------------|------------|-------------| |-------------------------------------------------------------------------------------------------------------------|-------------|----------------|------------|------------|-----------------------------------------------------------------------------------------------------|
| [Project Setup & Repository](https://github.com/dat515-2025/Group-8#) | Lukas | ✅ Complete | [X hours] | Medium | [Any notes] | | [Project Setup & Repository](https://github.com/dat515-2025/Group-8#) | Lukas | ✅ Complete | [X hours] | Medium | [Any notes] |
| [Design Document](https://github.com/dat515-2025/Group-8/blob/main/6design/design.md) | Both | ✅ Complete | 4 Hours | Easy | [Any notes] | | [Design Document](https://github.com/dat515-2025/Group-8/blob/main/6design/design.md) | Both | ✅ Complete | 4 Hours | Easy | [Any notes] |
| [Backend API Development](https://github.com/dat515-2025/Group-8/tree/main/7project/backend/app/api) | Dejan | ✅ Complete | 12 hours | Medium | [Any notes] | | [Backend API Development](https://github.com/dat515-2025/Group-8/tree/main/7project/backend/app/api) | Dejan | ✅ Complete | 12 hours | Medium | [Any notes] |
| [Database Setup & Models](https://github.com/dat515-2025/Group-8/tree/main/7project/backend/app/models) | Lukas | 🔄 In Progress | [X hours] | Medium | [Any notes] | | [Database Setup & Models](https://github.com/dat515-2025/Group-8/tree/main/7project/backend/app/models) | Lukas | ✅ Complete | [X hours] | Medium | [Any notes] |
| [Frontend Development](https://github.com/dat515-2025/Group-8/tree/main/7project/frontend) | Dejan | ✅ Complete | 17 hours | Medium | [Any notes] | | [Frontend Development](https://github.com/dat515-2025/Group-8/tree/main/7project/frontend) | Dejan | ✅ Complete | 17 hours | Medium | [Any notes] |
| [Docker Configuration](https://github.com/dat515-2025/Group-8/blob/main/7project/compose.yml) | Lukas | ✅ Complete | [X hours] | Easy | [Any notes] | | [Docker Configuration](https://github.com/dat515-2025/Group-8/blob/main/7project/compose.yml) | Lukas | ✅ Complete | 3 hours | Easy | [Any notes] |
| [Cloud Deployment](https://github.com/dat515-2025/Group-8/blob/main/7project/deployment/app-demo-deployment.yaml) | Lukas | ✅ Complete | [X hours] | Hard | [Any notes] | | [Cloud Deployment](https://github.com/dat515-2025/Group-8/blob/main/7project/deployment/app-demo-deployment.yaml) | Lukas | ✅ Complete | [X hours] | Hard | Using Talos cluster running in proxmox - easy snapshots etc. Frontend deployed at Cloudflare pages. |
| [Testing Implementation](https://github.com/dat515-2025/group-name) | Dejan | ✅ Complete | 16 hours | Medium | [Any notes] | | [Testing Implementation](https://github.com/dat515-2025/group-name) | Dejan | ✅ Complete | 16 hours | Medium | [Any notes] |
| [Documentation](https://github.com/dat515-2025/group-name) | Both | 🔄 In Progress | [X hours] | Easy | [Any notes] | | [Documentation](https://github.com/dat515-2025/group-name) | Both | 🔄 In Progress | [X hours] | Easy | [Any notes] |
| [Presentation Video](https://github.com/dat515-2025/group-name) | Both | ❌ Not Started | [X hours] | Medium | [Any notes] | | [Presentation Video](https://github.com/dat515-2025/group-name) | Both | ❌ Not Started | [X hours] | Medium | [Any notes] |
@@ -575,14 +576,27 @@ curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/authenticated-route
### [Lukáš] ### [Lukáš]
| Date | Activity | Hours | Description | ## Hour Sheet
|----------------|---------------------|------------|----------------------------------------------------|
| 4.10 to 10.10 | Initial Setup | 40 | Repository setup, project structure, cluster setup | **Name:** Lukáš Trkan
| 14.10 to 16.10 | Backend Development | 12 | Implemented user authentication - oauth |
| 8.10 to 12.10 | CI/CD | 10 | Created database schema and models | | Date | Activity | Hours | Description | Representative Commit / PR |
| [Date] | Testing | [X.X] | Unit tests for API endpoints | |:----------------|:----------------------------|:--------|:------------------------------------------------------------------------------------|:------------------------------------------------------|
| [Date] | Documentation | [X.X] | Updated README and design doc | | 18.9. - 19.9. | Initial Setup & Design | 40 | Repository init, system design diagrams, basic Terraform setup | `feat(infrastructure): add basic terraform resources` |
| **Total** | | **[XX.X]** | | | 20.9. - 5.10. | Core Infrastructure & CI/CD | 12 | K8s setup (ArgoCD), CI/CD workflows, RabbitMQ, Redis, Celery workers, DB migrations | `PR #2`, `feat(infrastructure): add rabbitmq cluster` |
| 6.10. - 9.10. | Frontend Infra & DB | 5 | Deployed frontend to Cloudflare, setup metrics, created database models | `PR #16` (Cloudflare), `PR #19` (DB structure) |
| 10.10. - 11.10. | Backend | 5 | Implemented OAuth support (MojeID, BankID) | `feat(auth): add support for OAuth and MojeID` |
| 12.10. | Infrastructure | 2 | Added database backups | `feat(infrastructure): add backups` |
| 16.10. | Infrastructure | 4 | Implemented secrets management, fixed deployment/env variables | `PR #29` (Deployment envs) |
| 17.10. | Monitoring | 1 | Added Sentry logging | `feat(app): add sentry loging` |
| 21.10. - 22.10. | Backend | 8 | Added ČSAS bank connection | `PR #32` (Fix React OAuth) |
| 29.10. - 30.10. | Backend | 5 | Implemented transaction encryption, add bank scraping | `PR #39` (CSAS Scraping) |
| 30.10. | Monitoring | 6 | Implemented Loki logging and basic Prometheus metrics | `PR #42` (Prometheus metrics) |
| 9.11. | Monitoring | 2 | Added custom Prometheus metrics | `PR #46` (Prometheus custom metrics) |
| 11.11. | Tests | 1 | Investigated and fixed broken Pytest environment | `fix(tests): set pytest env` |
| 11.11. - 12.11. | Features & Deployment | 6 | Added cron support, email sender service, updated workers & image | `PR #49` (Email), `PR #50` (Update workers) |
| 18.9 - 14.11 | Documentation | 8 | Updated report.md, design docs, and tfvars.example | `Create design.md`, `update report` |
| **Total** | | **105** | | |
### Dejan ### Dejan
@@ -610,16 +624,32 @@ curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/authenticated-route
[Reflect on the key technical and collaboration skills learned during this project] [Reflect on the key technical and collaboration skills learned during this project]
### Challenges Faced ### Challenges Faced
#### Slow cluster performance
This was caused by single SATA SSD disk running all VMs. This was solved by adding second NVMe disk just for Talos VMs.
[Describe the main challenges and how you overcame them] [Describe the main challenges and how you overcame them]
### If We Did This Again ### If We Did This Again
#### Different framework
FastAPI lacks usable build in support for database migrations and implementing Alembic was a bit tricky.
Tricky was also integrating FastAPI auth system with React frontend, since there is no official project template.
Using .NET (which we considered initially) would probably solve these issues.
[What would you do differently? What worked well that you'd keep?] [What would you do differently? What worked well that you'd keep?]
### Individual Growth ### Individual Growth
#### [Team Member 1 Name] #### [Lukas]
This course finally forced me to learn kubernetes (been on by TODO list for at least 3 years).
I had some prior experience with terraform/opentofu from work but this improved by understanding of it.
The biggest challenge for me was time tracking since I am used to tracking to projects, not to tasks.
(I am bad even at that :) ).
It was also interesting experience to be the one responsible for the initial project structure/design/setup
used not only by myself.
[Personal reflection on growth, challenges, and learning] [Personal reflection on growth, challenges, and learning]