PotatoMaxwell/mxpic_EDA
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

database removed outside

Browse Source
This commit is contained in:
PotatoMaxwell
2026-06-08 14:48:14 +08:00
parent bf8e72f5b6
commit 75dd78aa33
68 changed files with 679 additions and 1815 deletions
Download Patch File Download Diff File Expand all files Collapse all files
develop_plan/findings.md
+108
View File
@@ -0,0 +1,108 @@
# Project Findings
## Repository Inventory
- Static/frontend files are under `frontend/`: `login.html`, `dashboard.html`, `canvas.html`, and `canvas-helpers.js`.
- Backend files are under `backend/`: `server.py`, `database.py`, `pdk_access.py`, `technology_manifest.py`, `gds_builder.py`, routing helpers, icons, and `directories.yaml`.
- Persistent data appears under `database/`, including `mxpic_data.db`, saved layout YAML/SVG files, and generated `_exports/*.gds`.
- Tests are JavaScript static/unit-style tests under `tests/`.
- Project docs exist under `docs/`, including intranet deployment, GDS/SVG generation, and PDK technology loading notes.
- A checked saved layout example at `database/engineer/layout/mxpic_project_1/mxpic_project_1.yml` contains unresolved merge-conflict markers, so at least some local saved data may be invalid until cleaned.
- Conflict-marker search also found real markers in `database/engineer/layout/mxpic_project_1/mxpic_project_1.svg`, `tests/layout-ui-wiring.test.js`, and `tests/layout-backend-static.test.js`.
- `backend/directories.yaml` is a legacy/declarative directory category file with PDK-style categories; the active docs/source show library scanning is now driven by the role-scoped PDK filesystem.
## Saved Layout YAML Shape
- Saved layout YAML uses schema version `2.0.0`, `kind: cell`, `coordinate_system: gds_y_up`, `canvas_size`, `project`, `name`, `type`, and `version`.
- Modern YAML uses `pins`, `instances`, `elements`, and `bundles`.
- `instances` can reference basic components like `circle`, generated/forge components, other project cells, or PDK paths such as `Silterra/<technology>/primitives/...`.
- `bundles.output_bus.links` stores routed connections with source/target pins and route settings.
- Both checked `.project.json` metadata files (`admin` and `engineer`) select technology `Silterra/EMO1_2ML_CU_Al_RDL`.
## README / Deployment
- The project is `mxpic_EDA`, an EDA layout tool for OptiHK.
- Runtime requirement explicitly listed in README is Flask.
- The application can launch login, dashboard, canvas editing, YAML generation, and PDK browsing without build-time router dependencies.
- Build/export actions require `mxpic_router` and Nazca; if `mxpic_forge.Route` is unavailable, routing falls back to Nazca interconnects.
- SVG preview generation for routed/build output requires `gdstk`.
- Intranet server is started with `run_intranet_server.ps1` and listens on `0.0.0.0:3000` by default.
- LAN users browse to `http://<host-computer-ip>:3000`; firewall must allow inbound TCP `3000`.
- Default local accounts are `admin / 123456` and `engineer / 123456`.
- User project files are stored under `database/<username>/layout`.
- Important env vars include `MXPIC_SECRET_KEY`, `MXPIC_HOST`, `MXPIC_PORT`, `MXPIC_DEBUG`, `MXPIC_COOKIE_SECURE`, `MXPIC_PDK_PUBLIC_ROOT`, and `MXPIC_PDK_ATLAS_ROOT`.
- Public/developer PDK roots default to `../opt_pdk_public/foundries`; manager/atlas roots default to `../opt_pdk_atlas/foundries`.
- `run_intranet_server.ps1` sets default `MXPIC_SECRET_KEY` if absent, sets `MXPIC_HOST=0.0.0.0`, `MXPIC_PORT=3000`, `MXPIC_DEBUG=0`, then runs `python backend\server.py`.
- `backend/server.py` initializes the local SQLite database when the server process starts.
- Flask runs threaded on the configured host and port.
## GDS / SVG Generation
- `Build Layout` serializes the active canvas page into layout YAML, posts it to `/api/save-layout`, writes `database/<username>/layout/<project>/<cell>.yml`, writes optional route sidecar data, then calls router-backed SVG preview generation.
- Routed SVG preview uses `backend/routed_layout_preview.py`, which delegates to `mxpic_router.build_project_gds`, reads the temporary GDS with `gdstk`, and writes `<cell>.svg`.
- `Build GDS` posts to `/api/build-gds` with only the project name. It does not automatically serialize unsaved in-memory canvas state first.
- Final GDS export reads existing saved project YAML files, calls `backend/gds_builder.py`, delegates to `mxpic_router`, writes `database/_exports/<uuid>/<project>.gds`, returns a download URL, then cleans up the export folder after response close.
- Saved layout files are under `database/<username>/layout/<project>/`.
- External build dependencies are lazy: normal web usage can work without router stack, while preview/build actions require `mxpic_router`, Nazca, and sometimes `gdstk`.
## PDK / Technology / Xsections
- PDK components, technology manifests, and GDS assets all come from role-scoped PDK roots.
- Dashboard loads available technologies from `/api/technologies`, scanning `<active_role_pdk_root>/<foundry>/<technology>/technology.yml`.
- Project creation stores selected technology in `database/<username>/layout/<project>/.project.json`.
- Canvas loads project metadata from `/api/projects/<project>`, then loads `/api/technologies/<foundry>/<technology>/manifest`.
- `technology.yml` supplies layer mappings, routing type options, defaults, and xsection definitions such as `strip`, `rib_low`, `metal_1`, and `metal_2`.
- Canvas route editing uses the active technology manifest to populate xsection choices and route defaults.
- Saved YAML stores component paths and per-link route data such as `xsection`, `family`, `width`, `radius`, and `routing_type`.
- Build-time router applies `technology.yml` to Nazca layers/xsections before resolving PDK assets and routing links.
## Backend Route Surface
- `backend/server.py` serves `/`, `/dashboard`, `/canvas`, `/canvas-helpers.js`, and `/logout`.
- Login endpoint is `POST /login`.
- Health check endpoint is `/api/health`.
- Account/profile endpoints include `/api/profile`, `/api/profile/password`, and `/api/logs`.
- Project endpoints include listing/creating `/api/projects`, reading/deleting `/api/projects/<project_name>`, cell rename/delete under `/api/projects/<project_name>/cells/<cell_name>`, and `/api/save-layout`.
- Build/export endpoints include `/api/build-gds`, `/api/exports/<export_id>/<filename>`, and `/api/projects/<project_name>/gds/<filename>`.
- PDK/technology endpoints include `/api/technologies`, `/api/technologies/<foundry>/<technology>/manifest`, `/api/library`, `/api/component/<component_name>`, `/api/component/<component_name>/image`, and `/api/icon/<category>`.
- Project listing treats each `.yml`/`.yaml` inside a project folder as one saved cell/canvas.
- Project creation sanitizes the requested name, adds a numeric suffix if needed, creates an initial project folder, writes `.project.json`, and logs `project.create`.
- `/api/save-layout` accepts `project`, `cell`, `content`, and optional `preview` flag.
- `/api/library` rebuilds the component tree on each request so project technology and user role are reflected immediately.
## Frontend Page Flow
- `frontend/login.html` renders a login form that submits `POST /login` and then moves authenticated users into the dashboard.
- `frontend/dashboard.html` loads recent logs from `/api/logs`, profile data from `/api/profile`, projects from `/api/projects`, and technology choices from `/api/technologies`.
- Opening a project navigates to `/canvas?project=<name>`.
- Creating a project posts `{ name, technology }` to `/api/projects`.
- Dashboard also supports deleting projects, updating occupation, and changing password through backend APIs.
- `frontend/canvas.html` loads project metadata/cells from `/api/projects/<project>`, then loads the selected technology manifest.
- Canvas loads the component library with `/api/library?project=<project>`, component metadata from `/api/component/<component_name>?project=<project>`, and component images from `/api/component/<component_name>/image?project=<project>`.
- Canvas save/build controls call `/api/save-layout` and `/api/build-gds`.
- Dashboard `openProject(name)` navigates directly to `/canvas?project=<name>`.
- Canvas `loadTechnologyManifest` falls back to `FALLBACK_TECHNOLOGY_MANIFEST` when the saved technology is missing/invalid.
- Canvas `fetchLibrary` is project-scoped, so the visible library follows the selected project technology.
- Canvas source comments define the control split:
- `handleBuildLayout`: save active page, generate preview assets, show preview tab.
- `handleSaveProjectLayouts`: save YAML for every editable page without opening previews.
- `handleBuildGds`: build project GDS through the backend and trigger download.
## Database
- `backend/database.py` uses SQLite at `database/mxpic_data.db`.
- Tables: `users` and `user_logs`.
- User records include password hash, creation date, credits, occupation, and `user_group`.
- Database migrations add missing columns to older local SQLite files.
- Default account roles are `admin -> manager`, `engineer -> developers`, and ordinary users default to `user`.
- Audit logs record user action, project/cell context, detail, IP address, and UTC timestamp.
## PDK Access and Build Helpers
- `backend/pdk_access.py` defines supported user groups: `manager`, `developers`, and `user`.
- Managers resolve to `MXPIC_PDK_ATLAS_ROOT` or `opt_pdk_atlas/foundries`; developers/users resolve to `MXPIC_PDK_PUBLIC_ROOT` or `opt_pdk_public/foundries`.
- Only managers prefer full component GDS assets; other roles use public/black-box style assets.
- Temporary GDS exports are created in UUID folders and old export folders are cleaned up after an age threshold.
- `backend/gds_builder.py` validates that saved cell YAML files exist, then delegates all final GDS work to `mxpic_router.build_project_gds`.
- Local GDS building is not implemented as a fallback in this repository.
## Tests
- Tests are JavaScript assertion scripts in `tests/`.
- The test suite heavily checks static frontend/backend integration contracts, such as route existence, API strings, layout/GDS wiring, technology manifest loading, and PDK access behavior.
- `tests/canvas-helpers.test.js` directly imports `frontend/canvas-helpers.js` and checks handle placement, YAML serialization, route defaults, element ports, route YAML, and route crossing helpers.
- Several tests intentionally assert that legacy local preview/registry modules are absent and that the router-backed preview/GDS path is present.
- Current checked test files include unresolved conflict markers in at least two files, so `node` test execution may fail before assertions run unless those files are cleaned.
## Open Questions
- None remaining for the requested project usage/workflow summary.
develop_plan/progress.md
+50
View File
@@ -0,0 +1,50 @@
# Project Understanding Progress
## Session Log
- Started project read-through on 2026-06-08.
- Loaded requested `planning-with-files` skill and project-analysis guidance.
- Collected initial file inventory with `rg --files`.
- Created `develop_plan` planning notes.
- Read `README.md` and `docs/INTRANET_DEPLOYMENT.md`.
- Read `docs/GDS_SVG_GENERATION_LOGIC.md` and `docs/PDK_TECHNOLOGY_XSECTION_LOADING.md`.
- Listed `backend/server.py` route/function signatures.
- Read `backend/database.py`.
- Read `backend/pdk_access.py` and `backend/gds_builder.py`.
- Read `run_intranet_server.ps1` and backend startup/config references.
- Inspected login/dashboard and canvas frontend API wiring.
- Read dashboard and canvas handler context around project open, technology load, library load, save, preview, and GDS build.
- Read backend handler context and test structure.
- Read a saved engineer project YAML example and `backend/directories.yaml`.
- Searched for conflict markers and read admin/engineer `.project.json` metadata files.
- Wrote `develop_plan/project_understanding.md`.
## Commands / Checks
- `rg --files` succeeded with escalated approval after sandbox spawn failure.
- `Get-Content -Raw README.md` succeeded.
- `Get-Content -Raw docs\INTRANET_DEPLOYMENT.md` succeeded.
- `Get-Content -Raw docs\GDS_SVG_GENERATION_LOGIC.md` succeeded.
- `Get-Content -Raw docs\PDK_TECHNOLOGY_XSECTION_LOADING.md` succeeded.
- `rg "@app\\.route|def " backend\server.py` succeeded.
- `Get-Content -Raw backend\database.py` succeeded.
- `Get-Content -Raw backend\pdk_access.py` succeeded.
- `Get-Content -Raw backend\gds_builder.py` succeeded.
- `Get-Content -Raw run_intranet_server.ps1` succeeded.
- `rg -n "app.run|MXPIC_HOST|MXPIC_PORT|MXPIC_DEBUG|SECRET_KEY|init_db|database" backend\server.py` succeeded.
- `rg -n "fetch\\(|window\\.location|..." frontend\login.html frontend\dashboard.html` succeeded.
- `rg -n "fetch\\(|handleBuildLayout|handleBuildGds|..." frontend\canvas.html frontend\canvas-helpers.js` succeeded.
- `rg -n -C 4 "function openProject|..." frontend\dashboard.html` succeeded.
- `rg -n -C 5 "const loadProject|..." frontend\canvas.html` succeeded.
- `rg -n -C 8 "def login|def list_projects|..." backend\server.py` succeeded.
- `rg -n "node:test|assert|..." tests` succeeded.
- `Get-Content -Raw database\engineer\layout\mxpic_project_1\mxpic_project_1.yml` succeeded.
- `Get-Content -Raw backend\directories.yaml` succeeded.
- `rg -n "<<<<<<<|=======|>>>>>>>"` succeeded.
- `rg --files -g .project.json -g !database\_exports` succeeded.
- `Get-Content -Raw database\admin\layout\mxpic_project_1\.project.json` succeeded.
- `Get-Content -Raw database\engineer\layout\mxpic_project_1\.project.json` succeeded.
## Files Created / Modified
- Created `develop_plan/task_plan.md`.
- Created `develop_plan/findings.md`.
- Created `develop_plan/progress.md`.
- Created `develop_plan/project_understanding.md`.
develop_plan/project_understanding.md
+333
View File
@@ -0,0 +1,333 @@
# mxpic_EDA Project Understanding
## Purpose
`mxpic_EDA` is a local/intranet EDA web application for creating photonic integrated circuit layout projects. It combines:
- A Flask backend for authentication, project APIs, PDK browsing, layout persistence, preview generation, and GDS export.
- Static frontend pages for login, dashboard, and canvas editing.
- SQLite account/audit storage plus filesystem-based project layout storage.
- External build-time tooling (`mxpic_router`, Nazca, and sometimes `gdstk`) for routed SVG preview and final GDS generation.
The project is designed so normal login, dashboard, canvas editing, YAML generation, and PDK browsing can run without importing the router stack. Build actions load those heavier dependencies only when needed.
## Main Entry Points
### Start the App
The intended intranet startup path is:
```powershell
.\run_intranet_server.ps1
```
That script sets:
- `MXPIC_HOST=0.0.0.0`
- `MXPIC_PORT=3000`
- `MXPIC_DEBUG=0`
- a fallback `MXPIC_SECRET_KEY` if none is set
Then it runs:
```powershell
python backend\server.py
```
The server listens on:
```text
http://<host-computer-ip>:3000
```
Default local accounts are:
```text
admin / 123456
engineer / 123456
```
`admin` is a manager account. `engineer` is a developer account.
## Repository Structure
```text
backend/
server.py Flask app, routes, project/layout APIs
database.py SQLite initialization, users, profiles, audit logs
pdk_access.py Role-aware PDK root selection and export helpers
gds_builder.py Final GDS wrapper around mxpic_router
routed_layout_preview.py SVG preview generation via temporary routed GDS
router_dependency.py Lazy build-time dependency validation
technology_manifest.py technology.yml loading/validation
icons/ Category icons served by backend
frontend/
login.html Login page
dashboard.html Project/account dashboard
canvas.html Main layout editor
canvas-helpers.js Canvas/YAML/route helper functions and tests target
database/
mxpic_data.db SQLite app database
<username>/layout/... Saved project metadata, YAML cells, SVG previews
_exports/<uuid>/... Temporary generated GDS downloads
docs/
INTRANET_DEPLOYMENT.md
GDS_SVG_GENERATION_LOGIC.md
PDK_TECHNOLOGY_XSECTION_LOADING.md
tests/
JavaScript assertion tests for static wiring and canvas helpers
```
## User Workflow
1. User opens `/`.
2. If not logged in, Flask serves `frontend/login.html`.
3. Login form posts credentials to `POST /login`.
4. On success, the session stores user id, username, and user group.
5. User lands on `/dashboard`.
6. Dashboard loads:
- `/api/profile`
- `/api/logs`
- `/api/projects`
- `/api/technologies`
7. User creates or opens a project.
8. Creating a project posts `{ name, technology }` to `POST /api/projects`.
9. Opening a project navigates to `/canvas?project=<project>`.
10. Canvas loads project data from `/api/projects/<project>`.
11. Canvas loads the selected `technology.yml` manifest from `/api/technologies/<foundry>/<technology>/manifest`.
12. Canvas loads the project-scoped PDK library from `/api/library?project=<project>`.
13. User places components, elements, pins, and routes.
14. Save or Build Layout writes YAML to disk through `/api/save-layout`.
15. Build Layout also generates and opens an SVG preview.
16. Build GDS calls `/api/build-gds` and downloads the generated GDS.
## Project and Data Model
Each user's projects live under:
```text
database/<username>/layout/<project>/
```
Project metadata is stored in:
```text
database/<username>/layout/<project>/.project.json
```
Example:
```json
{
"name": "mxpic_project_1",
"technology": "Silterra/EMO1_2ML_CU_Al_RDL"
}
```
Saved cells are YAML files:
```text
database/<username>/layout/<project>/<cell>.yml
```
Generated previews are SVG files:
```text
database/<username>/layout/<project>/<cell>.svg
```
Modern saved layout YAML contains:
- `schema_version`
- `kind`
- `coordinate_system`
- `canvas_size`
- `project`
- `name`
- `type`
- `version`
- `pins`
- `instances`
- `elements`
- `bundles`
`instances` reference basic components, forge/generated components, other project cells, or PDK component paths. `bundles` store routed links and route settings such as `xsection`, `family`, `width`, `radius`, and `routing_type`.
## PDK and Technology Workflow
The active PDK root is chosen by user role:
- `manager`: `MXPIC_PDK_ATLAS_ROOT` or `opt_pdk_atlas/foundries`
- `developers` and `user`: `MXPIC_PDK_PUBLIC_ROOT` or `opt_pdk_public/foundries`
The dashboard lists available technologies by scanning:
```text
<active_role_pdk_root>/<foundry>/<technology>/technology.yml
```
The selected technology is saved in `.project.json`.
Canvas then uses that selected technology to:
- Load route defaults and xsections from `technology.yml`.
- Scope the component library browser to the chosen foundry/technology.
- Save component paths that can later resolve under the base role PDK root.
At build time, the router receives the active PDK root and the selected project's technology manifest path. The router applies technology layers/xsections to Nazca before resolving PDK assets and routing links.
## Build Layout vs Save vs Build GDS
### Save
The canvas "Save" path calls `handleSaveProjectLayouts`. It writes YAML for every editable project/cell page to `/api/save-layout` with preview disabled. It is for persistence, not preview.
### Build Layout
The canvas "Build Layout" path:
1. Validates route crossings.
2. Serializes the active page to YAML.
3. Posts to `/api/save-layout`.
4. Backend writes `<cell>.yml`.
5. Backend writes route sidecar data when available.
6. Backend calls `routed_layout_preview.py`.
7. `routed_layout_preview.py` calls `mxpic_router.build_project_gds`.
8. The temporary routed GDS is converted to `<cell>.svg` with `gdstk`.
9. Frontend opens the SVG preview tab.
### Build GDS
The canvas "Build GDS" path:
1. Validates route crossings across editable pages.
2. Posts `{ project }` to `/api/build-gds`.
3. Backend reads saved YAML files already on disk.
4. Backend creates `database/_exports/<uuid>/<project>.gds`.
5. `backend/gds_builder.py` delegates to `mxpic_router.build_project_gds`.
6. Backend returns `download_url`.
7. Frontend triggers a browser download.
8. Backend removes the temporary export folder after serving the file.
Important: Build GDS does not serialize unsaved in-memory canvas state first. Users should Save or Build Layout before Build GDS if they want latest canvas edits included.
## Backend API Map
Primary pages:
- `GET /`
- `POST /login`
- `GET /dashboard`
- `GET /canvas`
- `GET /canvas-helpers.js`
- `GET /logout`
Account and activity:
- `GET /api/health`
- `GET/PATCH /api/profile`
- `POST /api/profile/password`
- `GET/POST /api/logs`
Projects and cells:
- `GET /api/projects`
- `POST /api/projects`
- `GET /api/projects/<project_name>`
- `DELETE /api/projects/<project_name>`
- `PATCH/DELETE /api/projects/<project_name>/cells/<cell_name>`
- `POST /api/save-layout`
- `GET /api/projects/<project_name>/cells/<cell_name>/layout.svg`
Build/export:
- `POST /api/build-gds`
- `GET /api/exports/<export_id>/<filename>`
- `GET /api/projects/<project_name>/gds/<filename>`
PDK/technology/library:
- `GET /api/technologies`
- `GET /api/technologies/<foundry>/<technology>/manifest`
- `GET /api/library`
- `GET /api/component/<component_name>`
- `GET /api/component/<component_name>/image`
- `GET /api/icon/<category>`
## Persistence and Accounts
SQLite database:
```text
database/mxpic_data.db
```
Tables observed:
- `users`
- `user_logs`
The backend initializes the database on startup. It also performs lightweight migrations for profile, credit, occupation, and group fields.
User groups affect PDK access:
- `manager` can use atlas/private PDK roots and prefers full GDS assets.
- `developers` and `user` use the public PDK roots.
Audit logs record user actions with project/cell context, details, IP address, and UTC timestamp.
## Tests
Tests are JavaScript assertion scripts under `tests/`.
The suite mostly checks integration contracts by reading source files, including:
- API route strings and handler wiring.
- Build Layout and Build GDS frontend/backend wiring.
- Removal of legacy local preview/PDK registry modules.
- Router-backed GDS and SVG preview paths.
- Technology manifest and PDK access behavior.
- Canvas helper behavior such as pin/handle placement, YAML serialization, route defaults, route YAML, element ports, and route crossing checks.
Likely command:
```powershell
node --test tests
```
## Current Repository State Notes
This checkout contains unresolved merge-conflict markers in some tracked files:
- `database/engineer/layout/mxpic_project_1/mxpic_project_1.yml`
- `database/engineer/layout/mxpic_project_1/mxpic_project_1.svg`
- `tests/layout-ui-wiring.test.js`
- `tests/layout-backend-static.test.js`
That means some checked sample data may not parse as YAML/SVG, and the test suite may fail before reaching assertions until those files are cleaned.
The checked admin and engineer project metadata both select:
```text
Silterra/EMO1_2ML_CU_Al_RDL
```
## Mental Model
Think of this app as:
```text
Flask session/account layer
-> role-scoped PDK/technology selection
-> dashboard project metadata
-> canvas layout editing
-> YAML saved on disk
-> mxpic_router/Nazca build path
-> SVG preview or downloadable GDS
```
The repository itself owns the web UI, persistence, API contracts, and orchestration. The actual routed photonic layout build is intentionally delegated to the external `mxpic_router` stack.
develop_plan/task_plan.md
+19
View File
@@ -0,0 +1,19 @@
# Project Understanding Plan
## Goal
Read the `mxpic_EDA` project and document its usage and workflow in Markdown files under `develop_plan`.
## Phases
| Phase | Status | Notes |
|---|---|---|
| Initialize planning files | complete | Created `develop_plan` planning notes. |
| Inventory repository structure | complete | File list collected with `rg --files`. |
| Read README and operational docs | complete | README, intranet, GDS/SVG, and PDK technology docs read. |
| Read backend entry points and data layer | complete | Route surface, database, PDK access, and GDS builder inspected. |
| Read frontend entry points | complete | Login/dashboard/canvas API wiring and handler context inspected. |
| Summarize project usage/workflow | complete | Wrote `develop_plan/project_understanding.md`. |
## Errors Encountered
| Error | Attempt | Resolution |
|---|---|---|
| `windows sandbox: spawn setup refresh` | Tried normal PowerShell/read commands in sandbox | Re-ran required commands with escalated approval. |