lukas/uis-cloud-computing
1
0
Fork 0
mirror of https://github.com/dat515-2025/Group-8.git synced 2026-03-22 06:57:47 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Compare commits

base: lukas:db9092b78fd362370ac558df248babebe9e7c7d7
Branches Tags
lukas:main lukas:merge/core_simplificcation lukas:test_arm_build lukas:51-refactor-project-structure lukas:merge/update_workers lukas:merge/email_sender lukas:merge/prometheus_custom_metrics lukas:add_more_tests lukas:merge/prometheus_metrics lukas:merge/csas_scraping lukas:30-create-tests-and-set-up-a-github-pipeline lukas:merge/fix_react_oauth lukas:merge/frontend_basics lukas:merge/deployment_envs lukas:20-create-a-controller-layer-on-backend-side lukas:merge/database_backups lukas:merge/oauth lukas:merge/basic_database_structure lukas:11-update-deployment lukas:merge/cloudflare-deploy lukas:merge/pr_deploy_test lukas:merge/background-worker lukas:merge/refactor_migrations
...
compare: lukas:16f660ea5bfdf40d0ae257ced59426d237583ff7
Branches Tags
lukas:main lukas:merge/core_simplificcation lukas:test_arm_build lukas:51-refactor-project-structure lukas:merge/update_workers lukas:merge/email_sender lukas:merge/prometheus_custom_metrics lukas:add_more_tests lukas:merge/prometheus_metrics lukas:merge/csas_scraping lukas:30-create-tests-and-set-up-a-github-pipeline lukas:merge/fix_react_oauth lukas:merge/frontend_basics lukas:merge/deployment_envs lukas:20-create-a-controller-layer-on-backend-side lukas:merge/database_backups lukas:merge/oauth lukas:merge/basic_database_structure lukas:11-update-deployment lukas:merge/cloudflare-deploy lukas:merge/pr_deploy_test lukas:merge/background-worker lukas:merge/refactor_migrations

5 Commits
db9092b78f ... 16f660ea5b

Author SHA1 Message Date
ribardej
16f660ea5b feat(docs): finalized checklist.md 2025-11-16 21:07:16 +01:00
Lukáš Trkan
7a0d7dc4af update report 2025-11-16 17:30:51 +01:00
Lukáš Trkan
fabdff3bef update checklist 2025-11-16 17:27:13 +01:00
Lukáš Trkan
1c1130b9b0 update checklist 2025-11-16 17:21:45 +01:00
Lukáš Trkan
7d7698450d feat(deployment): Optimize Dockerfile 2025-11-16 17:07:32 +01:00
4 changed files with 136 additions and 64 deletions
Expand all files Collapse all files

119
7project/checklist.md
View File

@@ -7,65 +7,66 @@ Focus on areas that align with your project goals and interests.
The core deliverables are required. The core deliverables are required.
This means that you must get at least 2 points for each item in this category. This means that you must get at least 2 points for each item in this category.
| **Category** | **Item** | **Max Points** | **Points** | **Comment** | | **Category** | **Item** | **Max Points** | **Points** | **Comment** |
|:---------------------------------|:----------------------------------------|:---------------|:-----------|:----------------------------------------------------------------------------------------------------| |:---------------------------------|:----------------------------------------|:---------------|:-----------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Core Deliverables (Required)** | | | | | | **Core Deliverables (Required)** | | | | |
| Codebase & Organization | Well-organized project structure | 5 | 5 | Project is well-organized | | Codebase & Organization | Well-organized project structure | 5 | 5 | Project is well-organized, each part is separated (backend, frontend, IaC) and these parts are separated even mode (modules, packages...) |
| | Clean, readable code | 5 | 4 | Should be readable(function names should help), but readability can always be improved | | | Clean, readable code | 5 | 4 | Should be readable(function names should help), but readability can always be improved |
| | Use planning tool (e.g., GitHub issues) | 5 | 4 | We used Github issues | | | Use planning tool (e.g., GitHub issues) | 5 | 4 | We used Github issues |
| | Proper version control usage | 5 | 5 | We used branches for development, pull request reviews | | | Proper version control usage | 5 | 5 | We used branches for development, pull request reviews |
| 23 | Complete source code | 5 | 5 | The code is complete | | 23 | Complete source code | 5 | 5 | The code is complete - entire codebase is in this repository |
| Documentation | Comprehensive reproducibility report | 10 | 8 | Our report is precise | | Documentation | Comprehensive reproducibility report | 10 | 10 | Our report is precise, anybody should be able to reproduce our deployment by following provided instructions |
| | Updated design document | 5 | 4 | Our design document was updated and merged into the report | | | Updated design document | 5 | 4 | Our design document was updated and merged into the report |
| | Clear build/deployment instructions | 5 | 5 | Should be clear | | | Clear build/deployment instructions | 5 | 5 | Should be clear |
| | Troubleshooting guide | 5 | 3 | When it comes to troubleshooting, there is never enough documentation | | | Troubleshooting guide | 5 | 3 | When it comes to troubleshooting, there is never enough documentation |
| | Completed self-assessment table | 5 | 5 | Completed. | | | Completed self-assessment table | 5 | 5 | Completed. |
| 30 | Hour sheets for all members | 5 | 5 | Filled. | | 32 | Hour sheets for all members | 5 | 5 | Filled. |
| Presentation Video | Project demonstration | 5 | 5 | Yes | | Presentation Video | Project demonstration | 5 | 5 | Yes |
| | Code walk-through | 5 | 3 | There was not enough time to go through all of our code | | | Code walk-through | 5 | 3 | There was not enough time to go through all of our code, so we just mentioned some parts of it. |
| 13 | Deployment showcase | 5 | 5 | Yes | | 13 | Deployment showcase | 5 | 5 | Yes |
| **Technical Implementation** | | | | | | **Technical Implementation** | | | | |
| Application Functionality | Basic functionality works | 10 | 10 | The app works as intended | | Application Functionality | Basic functionality works | 10 | 10 | The app works as intended |
| | Advanced features implemented | 10 | 5 | OAuth, BankAPI connection (not only mock bank) | | | Advanced features implemented | 10 | 5 | OAuth, BankAPI connection (not only mock bank) |
| | Error handling & robustness | 10 | 5 | The app should be robust enough for normal use (TODO Lukasi kdyztak oprav pls) | | | Error handling & robustness | 10 | 5 | App notifies user about errors, errors in code are also logged by sentry and we get notified |
| 24 | User-friendly interface | 5 | 4 | Could always be better | | 24 | User-friendly interface | 5 | 4 | Responsive interface with dark mode support, should by user friendly enough |
| Backend & Architecture | Stateless web server | 5 | 0 | Our server is stateful | | Backend & Architecture | Stateless web server | 5 | 5 | Yes, the web server is stateless - authentication uses JWT, not sessions. |
| | Stateful application | 10 | 10 | Our server is stateful | | | Stateful application | 10 | 10 | Yes, the app is stateful - data are persistently stored in database |
| | Database integration | 10 | 10 | Yes, db is integrated | | | Database integration | 10 | 10 | We have deployed 3 MariaDB nodes with replication, MaxScale proxy and periodic backups. Connection app with this setup is same as with standalone db. |
| | API design | 5 | 5 | Backend APIs are implemented with public Swagger docs | | | API design | 5 | 5 | Backend APIs are implemented with public Swagger docs |
| 28 | Microservices architecture | 10 | 3 | Well not really but we have the app and the worker services divided (TODO Lukasi kdyztak oprav pls) | | 33 | Microservices architecture | 10 | 3 | We have separated API deployment and worker deployment. Worker process slow tasks - emails, payment scraping. There is no need for another service in current state but adding it is easy. |
| Cloud Integration | Basic cloud deployment | 10 | 10 | Yes (In private cluster) | | Cloud Integration | Basic cloud deployment | 10 | 10 | Yes (In private cluster), using GH Actions and self-hosted runner. |
| | Cloud APIs usage | 10 | 0 | Not Applicable (TODO Lukasi kdyztak oprav pls) | | | Cloud APIs usage | 10 | 8 | GH Actions deploys frontend to Cloudflare Pages, deployment creates CF tunnel record automatically |
| | Serverless components | 10 | 0 | No (TODO Lukasi kdyztak oprav pls) | | | Serverless components | 10 | 10 | We are using CF pages for frontend deployment |
| 10 | Advanced cloud services | 5 | 0 | Not Applicable (TODO Lukasi kdyztak oprav pls) | | 33 | Advanced cloud services | 5 | 5 | Using CF provides us with DDOS protection, access rules, it hides our IP |
| **DevOps & Deployment** | | | | | | **DevOps & Deployment** | | | | |
| Containerization | Basic Dockerfile | 5 | 5 | Yes | | Containerization | Basic Dockerfile | 5 | 5 | Yes |
| | Optimized Dockerfile | 5 | 0 | Not really (TODO Lukasi kdyztak oprav pls) | | | Optimized Dockerfile | 5 | 5 | Rootless Dockerfile |
| | Docker Compose | 5 | 5 | dev only | | | Docker Compose | 5 | 5 | For development environment |
| 15 | Persistent storage | 5 | 5 | Yes | | 20 | Persistent storage | 5 | 5 | Yes, using Longhorn. |
| Deployment & Scaling | Manual deployment | 5 | 5 | Yes, possible | | Deployment & Scaling | Manual deployment | 5 | 5 | Yes, possible by using Helm manually |
| | Automated deployment | 5 | 5 | Yes, withGithub runners | | | Automated deployment | 5 | 5 | Yes, with Github actions |
| | Multiple replicas | 5 | 5 | Yes (4) | | | Multiple replicas | 5 | 5 | Yes, 3 pods with API, 3 pods with workers, 3 database pods |
| 25 | Kubernetes deployment | 10 | 10 | Yes | | 25 | Kubernetes deployment | 10 | 10 | Yes |
| **Quality Assurance** | | | | | | **Quality Assurance** | | | | |
| Testing | Unit tests | 5 | 2 | Basic coverage | | Testing | Unit tests | 5 | 4 | All workflows are covered by tests |
| | Integration tests | 5 | 5 | Yes | | | Integration tests | 5 | 5 | Yes |
| | End-to-end tests | 5 | 5 | Yes | | | End-to-end tests | 5 | 5 | Yes |
| 12 | Performance testing | 5 | 0 | No | | 14 | Performance testing | 5 | 0 | No |
| Monitoring & Operations | Health checks | 5 | 5 | Yes | | Monitoring & Operations | Health checks | 5 | 5 | Yes |
| | Logging | 5 | 2 | only to terminal add logstash | | | Logging | 5 | 4 | Logs can be accessed easily using Grafana |
| | Metrics/Monitoring | 2 | 2 | Yes | | | Metrics/Monitoring | 2 | 2 | Yes, visualised in Grafana |
| 12 | Custom Metrics for your project | 3 | 3 | Prometheus, Grafana | | 14 | Custom Metrics for your project | 3 | 3 | Yes, API has /metrics endpoint providing information about FastAPI itself and custom information such as number of users or transactions. |
| Security | HTTPS/TLS | 5 | 5 | Yes | | Security | HTTPS/TLS | 5 | 5 | Yes |
| | Authentication | 5 | 5 | Yes | | | Authentication | 5 | 5 | Yes |
| 15 | Authorization | 5 | 5 | Yes | | 15 | Authorization | 5 | 5 | Yes |
| **Innovation & Excellence** | | | | | | **Innovation & Excellence** | | | | |
| Advanced Features and | AI/ML Integration | 10 | 0 | No | | Advanced Features and | AI/ML Integration | 10 | 0 | No |
| Technical Excellence | Real-time features | 10 | 0 | No | | Technical Excellence | Real-time features | 10 | 0 | No |
| | Creative problem solving | 10 | 4 | Cron jobs for bank scraping | | | Creative problem solving | 10 | 4 | Cron jobs for bank scraping |
| | Performance optimization | 5 | 3 | Delegating emails and scraping to workers | | | Performance optimization | 5 | 4 | Delegating emails and scraping to workers, hosting frontend on CF |
| 7 | Exceptional user experience | 5 | 0 | | | 11 | Exceptional user experience | 5 | 3 | |
| **Total** | | **255** | **214** | | | **Total** | | **255** | **257** | |
## Grading Scale ## Grading Scale
- **Minimum Required: 100 points** - **Minimum Required: 100 points**

10
7project/report.md
View File

@@ -47,6 +47,7 @@ flowchart TB
svc <-- HTTP request/response --> api[("UniRate API")] svc <-- HTTP request/response --> api[("UniRate API")]
client <-- HTTP request/response --> n2 client <-- HTTP request/response --> n2
``` ```
The workflow works in the following way: The workflow works in the following way:
- Client connects to the frontend. After login, frontend automatically fetches the stored transactions from - Client connects to the frontend. After login, frontend automatically fetches the stored transactions from
@@ -666,7 +667,7 @@ kubectl exec -it <pod-name> -n prod -- /bin/bash
| [Cloud Deployment](https://github.com/dat515-2025/Group-8/pull/16) | Both | ✅ Complete | 21 hours | Hard | | | [Cloud Deployment](https://github.com/dat515-2025/Group-8/pull/16) | Both | ✅ Complete | 21 hours | Hard | |
| [Testing Implementation](https://github.com/dat515-2025/Group-8/pull/31/) | Both | ✅ Complete | 21 hours | Medium | | | [Testing Implementation](https://github.com/dat515-2025/Group-8/pull/31/) | Both | ✅ Complete | 21 hours | Medium | |
| [Documentation](https://github.com/dat515-2025/Group-8/commit/515106b238bc032d5f7d5dcae931b5cb7ee2a281) | Both | ✅ Complete | 14 hours | Medium | | | [Documentation](https://github.com/dat515-2025/Group-8/commit/515106b238bc032d5f7d5dcae931b5cb7ee2a281) | Both | ✅ Complete | 14 hours | Medium | |
| [Presentation Video](https://github.com/dat515-2025/group-name) TODO | Both | Started | 3 hours | Medium | | | [Presentation Video](https://youtu.be/FKR85AVN8bI) | Both | ✅ Complete | 3 hours | Medium | |
## Hour Sheet ## Hour Sheet
@@ -687,8 +688,9 @@ kubectl exec -it <pod-name> -n prod -- /bin/bash
| 9.11. | Monitoring | 2 | Added custom Prometheus metrics | `PR #46` (Prometheus custom metrics) | | 9.11. | Monitoring | 2 | Added custom Prometheus metrics | `PR #46` (Prometheus custom metrics) |
| 11.11. | Tests | 1 | Investigated and fixed broken Pytest environment | `fix(tests): set pytest env` | | 11.11. | Tests | 1 | Investigated and fixed broken Pytest environment | `fix(tests): set pytest env` |
| 11.11. - 12.11. | Features & Deployment | 6 | Added cron support, email sender service, updated workers & image | `PR #49` (Email), `PR #50` (Update workers) | | 11.11. - 12.11. | Features & Deployment | 6 | Added cron support, email sender service, updated workers & image | `PR #49` (Email), `PR #50` (Update workers) |
| 18.9 - 14.11 | Documentation | 8 | Updated report.md, design docs, and tfvars.example | `Create design.md`, `update report` | | 18.9 - 16.11 | Documentation | 8 | Updated report.md, design docs, and tfvars.example | `Create design.md`, `update report` |
| **Total** | | **105** | | | | 15.11 | Video | 2 | Record my video part, edit video | |
| **Total** | | **107** | | |
### Dejan ### Dejan
@@ -710,7 +712,7 @@ kubectl exec -it <pod-name> -n prod -- /bin/bash
| 25.9. to 14.11. | Documentation | 8 | Documenting the dev process | multiple `feat(docs): report.md update` | | 25.9. to 14.11. | Documentation | 8 | Documenting the dev process | multiple `feat(docs): report.md update` |
| **Total** | | **87** | | | | **Total** | | **87** | | |
### Group Total: 192 hours ### Group Total: 194 hours
--- ---

10
7project/src/backend/Dockerfile
View File

@@ -1,8 +1,16 @@
FROM python:3.11-trixie FROM python:3.11-slim
WORKDIR /app WORKDIR /app
RUN useradd --create-home --shell /bin/bash app
COPY requirements.txt . COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt RUN pip install --no-cache-dir -r requirements.txt
COPY . . COPY . .
RUN chown -R app:app /app
USER app
EXPOSE 8000 EXPOSE 8000
CMD ["sh", "-c", "alembic upgrade head && uvicorn app.app:fastApi --host 0.0.0.0 --port 8000"] CMD ["sh", "-c", "alembic upgrade head && uvicorn app.app:fastApi --host 0.0.0.0 --port 8000"]

61
todo.md Normal file
View File

@@ -0,0 +1,61 @@
- sentry for error tracking
- github actions for ci/cd
- deployment using helmchart, prod + dev
- celery background tasks
- cron
- cloudflare tunnels
- mariadb operator for database, maxscale 3 nodes, phpmyadmin, maxscale ui, backups
- prometheus + grafana dashboards for monitoring
- rabbitmq message queue
- longhorn for file storage (edited talosos deployment)
Aplikace:
- umíme oauth - mojeid, bankid => oboje se používá i pro autorizaci komunikace s CZ státem
- automatický import plateb z banky
- ukázat frontend
- ukázat backend - oauth, celery tasky,
- alembic na migrace db
- sentry integrace
Infrastruktura:
- běžíme na proxmoxu
- kubernetes cluster přes TalosOS jako virtuálky with edited configs
- deployment služeb přes tofu/terraform
- longhorn pro persistentní storage
- mariadb operator s maxscale - 3 nody + phpmyadmin
- prometheus + grafana monitoring
- rabbitmq message queue pro background tasky
- veřejný přístup přes cloudflare tunnels
- ukázat headlamp map
- build v github actions
- deployment pomocí helmchartu prod + dev prostředí pomocí selfhosted runneru
- remote access do clusteru přes tailscale
-
Where to put description of infrastructure components? for example Helm chart of app is quite large
How about readme?
linking commits in time table? I cant link all of them - many of them was directly to main branch
- for example iterative CI/CD setup
done
create src in 7project
make the chart vertical
urls as hyperlinks text
frontned npm run dev as codeblock
describe folder structure in readme in 7project
add comment column to checklist
readme in src folder with folder structure description
readme in src/tofu
readme in src/charts
add link to report for app
write few lines in Build intructrions, deployment instructions like in testing
add other services in cluster in report architecture overview
debug commands - kubectl logs -f podname, mention grafana