diff --git a/agents.md b/agents.md index 3c71122..4c63bcb 100644 --- a/agents.md +++ b/agents.md @@ -5,9 +5,9 @@ Developer and user documentation site for Freeshard, published at docs.freeshard ## Tech Stack - **Generator**: MkDocs with Material theme -- **Plugins**: blog, glightbox (image lightbox) +- **Plugins**: glightbox (image lightbox) - **Markdown extensions**: admonitions, details, syntax highlighting, Mermaid diagrams, emoji, markdown-include -- **Dependencies**: `requirements.txt` (mkdocs, mkdocs-material, markdown-include, mkdocs-glightbox), pinned to exact versions — MkDocs 1.x is unmaintained and MkDocs 2.0 is not a viable upgrade (no plugin system, no Material support). Do not bump these; the migration target is Zensical, tracked in [issue #10](https://github.com/FreeshardBase/documentation/issues/10) and blocked on moving the blog to the landing page ([issue #4](https://github.com/FreeshardBase/documentation/issues/4)). +- **Dependencies**: `requirements.txt` (mkdocs, mkdocs-material, markdown-include, mkdocs-glightbox), pinned to exact versions — MkDocs 1.x is unmaintained and MkDocs 2.0 is not a viable upgrade (no plugin system, no Material support). Do not bump these; the migration target is Zensical, tracked in [issue #10](https://github.com/FreeshardBase/documentation/issues/10). That was blocked on the Material `blog` plugin, which Zensical does not implement; the blog has since moved to freeshard.net ([issue #4](https://github.com/FreeshardBase/documentation/issues/4)) and the plugin is gone, so #10 is unblocked. ## Commands @@ -30,10 +30,7 @@ docs/ includes/ Reusable markdown snippets (template vars, portal name info) img/ Developer docs images user_guides/ End-user guides (password management, smart home) - blog/ - posts/ Blog posts (date-prefixed directories) - YYYY-MM-DD_slug/ Each post: main.md + images - .authors.yml Blog author metadata + blog/ Only redirect stubs now — see Retired Blog below css/extra.css Custom styles img/ Shared images (logo) mkdocs.yml Site config, navigation, theme, plugins @@ -42,16 +39,22 @@ mkdocs.yml Site config, navigation, theme, plugins ## Conventions ### Navigation -Navigation is explicitly defined in `mkdocs.yml` under the `nav` key. Four top-level sections: Overview, Developer Docs, Blog, User Guides. Adding a new page requires adding it to `nav`. +Navigation is explicitly defined in `mkdocs.yml` under the `nav` key. Three top-level sections: Overview, Developer Docs, User Guides. Adding a new page requires adding it to `nav`. -### Blog Posts -Blog posts live in `docs/blog/posts/YYYY-MM-DD_slug/main.md`. Each post directory contains the markdown file and any associated images. Authors are defined in `docs/blog/.authors.yml`. +### Retired Blog +The blog no longer lives here. It moved to `freeshard.net//blog/`, as an Astro content collection in the `landing-page` repo — write new posts there, not here. + +What remains under `docs/blog/` is 20 hand-written `index.html` redirect stubs, one per URL the Material `blog` plugin used to publish (13 posts, 5 archive years, `page/2/`, and the index). MkDocs copies non-markdown files through verbatim, so they land at exactly the old paths. Each stub is an instant `meta refresh` plus a `rel=canonical` and a visible link. + +They are static HTML rather than a server-side 301 because `docs.freeshard.net` is served by GitHub Pages, which cannot issue arbitrary redirects. Google [documents](https://developers.google.com/search/docs/crawling-indexing/301-redirects) that it reads an instant `meta refresh` as a permanent redirect, while still recommending a server-side redirect where one is possible. Putting the site behind a proxy that can answer 301s is the upgrade path if the link equity ever proves to matter. + +Do not delete these stubs, and do not let a generator swap drop them — they are the only thing standing between the old URLs and a 404. ### Markdown Includes Reusable snippets in `docs/developer_docs/includes/` can be included in other docs via the `markdown-include` extension: `{!developer_docs/includes/snippet.md!}`. ### Images -Store images alongside the content that uses them (in subdirectory `img/` or in blog post directories). Use relative paths. The glightbox plugin automatically adds lightbox behavior to images. +Store images alongside the content that uses them (in a subdirectory `img/`). Use relative paths. The glightbox plugin automatically adds lightbox behavior to images. ### Mermaid Diagrams Supported via `pymdownx.superfences` custom fence. Use ` ```mermaid ` code blocks. diff --git a/docs/blog/.authors.yml b/docs/blog/.authors.yml deleted file mode 100644 index d0d05a3..0000000 --- a/docs/blog/.authors.yml +++ /dev/null @@ -1,5 +0,0 @@ -authors: - max: - name: Max von Tettenborn - description: Founder - avatar: https://avatars.githubusercontent.com/u/1144702?v=4 diff --git a/docs/blog/2022/03/02/getting-rid-of-registration-and-login-views/index.html b/docs/blog/2022/03/02/getting-rid-of-registration-and-login-views/index.html new file mode 100644 index 0000000..12ace68 --- /dev/null +++ b/docs/blog/2022/03/02/getting-rid-of-registration-and-login-views/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/getting-rid-of-registration-and-login-views/.

+ + diff --git a/docs/blog/2022/03/30/new-features-for-the-portal-app-store/index.html b/docs/blog/2022/03/30/new-features-for-the-portal-app-store/index.html new file mode 100644 index 0000000..030d51e --- /dev/null +++ b/docs/blog/2022/03/30/new-features-for-the-portal-app-store/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/new-features-for-the-portal-app-store/.

+ + diff --git a/docs/blog/2022/06/10/starting-and-stopping-apps/index.html b/docs/blog/2022/06/10/starting-and-stopping-apps/index.html new file mode 100644 index 0000000..ca9bb17 --- /dev/null +++ b/docs/blog/2022/06/10/starting-and-stopping-apps/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/starting-and-stopping-apps/.

+ + diff --git a/docs/blog/2022/06/28/shared-directories/index.html b/docs/blog/2022/06/28/shared-directories/index.html new file mode 100644 index 0000000..159ae82 --- /dev/null +++ b/docs/blog/2022/06/28/shared-directories/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/shared-directories/.

+ + diff --git a/docs/blog/2022/07/25/backup-your-portal/index.html b/docs/blog/2022/07/25/backup-your-portal/index.html new file mode 100644 index 0000000..1ca6327 --- /dev/null +++ b/docs/blog/2022/07/25/backup-your-portal/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/backup-your-portal/.

+ + diff --git a/docs/blog/2022/11/01/peer-2-peer-communication/index.html b/docs/blog/2022/11/01/peer-2-peer-communication/index.html new file mode 100644 index 0000000..b17879d --- /dev/null +++ b/docs/blog/2022/11/01/peer-2-peer-communication/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/peer-2-peer-communication/.

+ + diff --git a/docs/blog/2022/12/08/smart-home-and-iot/index.html b/docs/blog/2022/12/08/smart-home-and-iot/index.html new file mode 100644 index 0000000..cdba165 --- /dev/null +++ b/docs/blog/2022/12/08/smart-home-and-iot/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/smart-home-and-iot/.

+ + diff --git a/docs/blog/2023/09/16/app-integration-overhaul/index.html b/docs/blog/2023/09/16/app-integration-overhaul/index.html new file mode 100644 index 0000000..f9b1504 --- /dev/null +++ b/docs/blog/2023/09/16/app-integration-overhaul/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/app-integration-overhaul/.

+ + diff --git a/docs/blog/2024/04/07/eating-your-own-dog-food/index.html b/docs/blog/2024/04/07/eating-your-own-dog-food/index.html new file mode 100644 index 0000000..054518f --- /dev/null +++ b/docs/blog/2024/04/07/eating-your-own-dog-food/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/eating-your-own-dog-food/.

+ + diff --git a/docs/blog/2024/09/10/going-paperless-with-paperless-on-portal/index.html b/docs/blog/2024/09/10/going-paperless-with-paperless-on-portal/index.html new file mode 100644 index 0000000..22aed1c --- /dev/null +++ b/docs/blog/2024/09/10/going-paperless-with-paperless-on-portal/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/going-paperless-with-paperless-on-portal/.

+ + diff --git a/docs/blog/2025/04/29/rebrand-and-public-source-code/index.html b/docs/blog/2025/04/29/rebrand-and-public-source-code/index.html new file mode 100644 index 0000000..72a2587 --- /dev/null +++ b/docs/blog/2025/04/29/rebrand-and-public-source-code/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/rebrand-and-public-source-code/.

+ + diff --git a/docs/blog/2026/05/29/unblocking-apps-by-fixing-a-performance-bottleneck/index.html b/docs/blog/2026/05/29/unblocking-apps-by-fixing-a-performance-bottleneck/index.html new file mode 100644 index 0000000..4616c07 --- /dev/null +++ b/docs/blog/2026/05/29/unblocking-apps-by-fixing-a-performance-bottleneck/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/unblocking-apps-by-fixing-a-performance-bottleneck/.

+ + diff --git a/docs/blog/2026/07/25/putting-apps-to-sleep-and-waking-them-in-under-two-seconds/index.html b/docs/blog/2026/07/25/putting-apps-to-sleep-and-waking-them-in-under-two-seconds/index.html new file mode 100644 index 0000000..74ff8f6 --- /dev/null +++ b/docs/blog/2026/07/25/putting-apps-to-sleep-and-waking-them-in-under-two-seconds/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/putting-apps-to-sleep-and-waking-them-in-under-two-seconds/.

+ + diff --git a/docs/blog/archive/2022/index.html b/docs/blog/archive/2022/index.html new file mode 100644 index 0000000..475c7b4 --- /dev/null +++ b/docs/blog/archive/2022/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/.

+ + diff --git a/docs/blog/archive/2023/index.html b/docs/blog/archive/2023/index.html new file mode 100644 index 0000000..475c7b4 --- /dev/null +++ b/docs/blog/archive/2023/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/.

+ + diff --git a/docs/blog/archive/2024/index.html b/docs/blog/archive/2024/index.html new file mode 100644 index 0000000..475c7b4 --- /dev/null +++ b/docs/blog/archive/2024/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/.

+ + diff --git a/docs/blog/archive/2025/index.html b/docs/blog/archive/2025/index.html new file mode 100644 index 0000000..475c7b4 --- /dev/null +++ b/docs/blog/archive/2025/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/.

+ + diff --git a/docs/blog/archive/2026/index.html b/docs/blog/archive/2026/index.html new file mode 100644 index 0000000..475c7b4 --- /dev/null +++ b/docs/blog/archive/2026/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/.

+ + diff --git a/docs/blog/index.html b/docs/blog/index.html new file mode 100644 index 0000000..475c7b4 --- /dev/null +++ b/docs/blog/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/.

+ + diff --git a/docs/blog/index.md b/docs/blog/index.md deleted file mode 100644 index e69de29..0000000 diff --git a/docs/blog/page/2/index.html b/docs/blog/page/2/index.html new file mode 100644 index 0000000..c649747 --- /dev/null +++ b/docs/blog/page/2/index.html @@ -0,0 +1,14 @@ + + + + +Moved to freeshard.net + + + + + +

The Freeshard blog now lives on freeshard.net.

+

If you are not redirected automatically, follow https://freeshard.net/en/blog/2/.

+ + diff --git a/docs/blog/posts/2022-03-02_getting_rid_of_registration_and_login_views/main.md b/docs/blog/posts/2022-03-02_getting_rid_of_registration_and_login_views/main.md deleted file mode 100644 index 8b61b80..0000000 --- a/docs/blog/posts/2022-03-02_getting_rid_of_registration_and_login_views/main.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -draft: false -date: 2022-03-02 -authors: - - max ---- - -# Getting rid of Registration and Login Views - -We are continuously working on bringing more apps into our app store. But since we are using apps that were originally made for self-hosting, we often need to make some tweaks. In particular, we want the user to be able to open an app after installation without the need to go through a registration or login form. Each Portal is mainly a single-user platform so that would be quite confusing. - - - -Some apps can be configured with a proxy auth feature out of the box. It can often be enabled from an environment variable and causes the app to read the logged-in user from a http header. This works great if it is deployed behind a reverse proxy that handles authentication and sets the header - exactly the situation that we have on the Portal, so that is our favourite way of getting rid of confusing login screens. - -For the sake of simplicity, we want to be able to apply the needed configuration purely through settings in the app.json file, if possible. Setting environment variables has been possible for a long time. But configuring http headers that are added to all requests that are forwarded to the app was not possible. That has now changed and you now can specify these http headers in app.json. See [the updated documentation](https://docs.getportal.org/developer_docs/routing_and_ac/){target=_blank} for more information. - -In fact, we had to add another feature as a dependency: [app.json format versions](https://docs.getportal.org/developer_docs/app_json/#versioning){target=_blank}. Up until now, there was only one valid format for the app.json document at a time. But with the http header feature, we wanted to fundamentally change the way URL paths are configured. For that to go smoothly, we introduced version 1.0 of the app.json format. Portal still understands version 0.0 though and transparently converts those older documents to 1.0. - -We believe that this change will make it easier to adapt apps to the Portal app store and in particular to make the user experience smoother by getting rid of registration and login views. diff --git a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/app_page.png b/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/app_page.png deleted file mode 100644 index 7ea2406..0000000 Binary files a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/app_page.png and /dev/null differ diff --git a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/featured_apps.png b/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/featured_apps.png deleted file mode 100644 index d2da851..0000000 Binary files a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/featured_apps.png and /dev/null differ diff --git a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/full_app_json.png b/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/full_app_json.png deleted file mode 100644 index 4d70ba3..0000000 Binary files a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/full_app_json.png and /dev/null differ diff --git a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/hints_popup.png b/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/hints_popup.png deleted file mode 100644 index 1caf24a..0000000 Binary files a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/hints_popup.png and /dev/null differ diff --git a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/label.png b/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/label.png deleted file mode 100644 index 009183d..0000000 Binary files a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/label.png and /dev/null differ diff --git a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/main.md b/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/main.md deleted file mode 100644 index bbf36e6..0000000 --- a/docs/blog/posts/2022-03-30_new_features_for_the_portal_app_store/main.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -draft: false -date: 2022-03-30 -authors: - - max ---- - -# New features for the Portal App store - -⚠️ If you own a Portal, restart it now, and it will upgrade itself. (Use the settings menu on the top-right.) - -When we first designed the app store, we focused on building something that works quickly. It was always meant to be a first shot that would be extended or replaced at some time. We now felt compelled to address this task. - - - -After adding more apps over time, we observed that some can be integrated with Portal pretty well, while others cannot. A typical caveat is a registration/login flow that we cannot get rid of with configuration alone. We talked about that in the last post. Those not-so-well-integrated apps do work, however, and can be very useful. We do not want to keep them out of the store. - -What we needed was a way to signal to the user which apps provide the Portal experience like it is meant to be: easy to use right away, no setup, no registration or login. We needed a way to highlight featured apps. - -So that is what we implemented and while we were at it, we searched through the backlog for other ideas for improving the app store that we collected over time. The result is an app store that looks much more finished (although not quite). - -And in order to support the changes, we also introduced version 2.0 of the app.json format (see the docs [here](/developer_docs/app_json/#version-10-to-version-20){target=_blank} and [here](/developer_docs/submitting/#metadata-for-the-app-store){target=_blank}). Versioning of this format was also described in the last post, and it already turned out to be very useful, making changes very easy and straightforward. - -Here is the full list of changes. - -👉 Featured Apps allows us to highlight apps that work particularly well. They are also sorted at the top of the list. - -![Featured Apps](./featured_apps.png) - -👉 Short and long description let us display a short text in the app list and a more verbose one on the app page. - -👉 Speaking of which: the app page did not exist before. It is the place for more in-depth information about an app. We also put the Install button there. - -![App Page](./app_page.png) - -👉 For some apps, the app page also contains a hints popup. Here, we can post important information about an app's usage or behaviour. - -![Hints Popup](./hints_popup.png) - -👉 People that are curious about such things can display the full app.json of each app. - -![Full app.json](./full_app_json.png) - -👉 We also got rid of the tabbed view, where you could switch between the app store and the list of installed apps. Instead, we mark each installed app with a label and let you uninstall it right from its page. - -![Label](./label.png) - -While working on the improvements, we thought of lots of other things we would like to do but for which we could not take the time. So expect another future update. Also, tell us what you think and if there are other features that you would like to see. diff --git a/docs/blog/posts/2022-06-10_starting_and_stopping_apps/authforward.png b/docs/blog/posts/2022-06-10_starting_and_stopping_apps/authforward.png deleted file mode 100644 index 8391713..0000000 Binary files a/docs/blog/posts/2022-06-10_starting_and_stopping_apps/authforward.png and /dev/null differ diff --git a/docs/blog/posts/2022-06-10_starting_and_stopping_apps/errorpages.png b/docs/blog/posts/2022-06-10_starting_and_stopping_apps/errorpages.png deleted file mode 100644 index 5462e24..0000000 Binary files a/docs/blog/posts/2022-06-10_starting_and_stopping_apps/errorpages.png and /dev/null differ diff --git a/docs/blog/posts/2022-06-10_starting_and_stopping_apps/main.md b/docs/blog/posts/2022-06-10_starting_and_stopping_apps/main.md deleted file mode 100644 index d10e925..0000000 --- a/docs/blog/posts/2022-06-10_starting_and_stopping_apps/main.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -draft: false -date: 2022-06-10 -authors: - - max ---- - -# Starting and Stopping Apps - -## Rationale - -Running Apps on Portal is realized through docker-compose. When you install an app, the docker-compose.yml that contains all apps is rerendered and the process restarted. This causes docker-compose to pick up the new app and to pull it and start it. - -However, until recently that meant that every app you have installed is running all the time. This limited the number of installed apps severely as the bottleneck was the Portal's RAM - 1GB with the current setup. - - - -It was important to do something about that. And the most intuitive approach is starting apps only on demand, just like you are used to on your smartphone. For this to work, we implemented several new features that work hand in hand to make this possible while maintaining a smooth user experience. - -## Detecting App Usage - -In order to start an app on demand, Portal first must detect when it is being used. Luckily, that is straightforward. We were already using the [Traefik forward auth feature](https://doc.traefik.io/traefik/middlewares/http/forwardauth/){target=_blank} for determining access control. The Portal Core acts as the auth server and thus gets called for all requests to all apps. - -![Traefik Forward Auth](./authforward.png) - -## Starting and Stopping Apps - -Containers that are defined using docker-compose can still be started and stopped by issuing commands to the docker daemon. Nothing else does the Portal Core. - -Whenever a successful auth request for an app hits the Portal Core, it checks whether the app is running and if not, it issues a start command. It also saves the timestamp of the request and during a scheduled task that runs every few seconds, it looks for apps that have not been used in a while and stops them in the same way. - -## Splash Screens - -Now that apps are not running all the time, it might happen that the user opens an app by clicking on it on the home screen and then has to wait some time until that app has been started and is accepting requests. We somehow have to bridge that time in a way that feels natural. A common practice is showing a splash screen and so that is what we implemented. - -Of course, the splash screen cannot be part of the app itself because a) those are apps that we don't create ourselves and b) when the splash screen is needed, the app is not running yet. So the splash screen is hosted by Portal Core. It is a simple static document containing the app's logo (statically embedded as a data-URL) and its name. It also reloads itself every few seconds so that as soon as the app is running, the reload will open the app. - -On the backend, we have to implement a kind of routing. If the app is not reachable, we want to reply with the splash screen instead. Again, we are using a Traefik feature for this. The error page middleware is originally meant to display custom error pages but since the splash screen can be seen as a kind of error page, we can use it here, too. - -![Traefik Error Page](./errorpages.png) - -## Defining the Lifecycle - -With all these features in place, automatically starting and stopping apps is working well. What is missing is a way for app developers to configure the behavior. Some apps start quickly and can be stopped early, others should keep running longer. Some apps should never be stopped at all. - -The right place to define this is of course the app.json file. We have covered it in prevous posts and also explained the way in which its format can be versioned. Adding the lifecycle section resulted in [version 3.1](/developer_docs/app_json/#version-30-to-version-31){target=_blank}. It allows us to define the idle timeout after which an app is stopped or disable automatically stopping it at all. - -In the future, we would like to include a method to periodically start the app even if it is not actively used. Some apps like ChangeDetection would benefit from that. And even further in the future, we want to enable apps to control their own stop and start behaviour by calling an internal endpoint. This is of course only useful for apps that are specifically created or adapted to Portal and we don't have those yet. - -Are there any other features regarding app lifecycle management that you would like to see? Write in the comments or via email. Or even better: use our [new feedback tool](https://feedback.getportal.org/){target=_blank}! diff --git a/docs/blog/posts/2022-06-28_shared_directories/main.md b/docs/blog/posts/2022-06-28_shared_directories/main.md deleted file mode 100644 index a9affac..0000000 --- a/docs/blog/posts/2022-06-28_shared_directories/main.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -draft: false -date: 2022-06-28 -authors: - - max ---- - -# Shared Directories - -⚠️ This feature is not backward-compatible. Your current Portal must be deleted, and you need to recreate it. [Write](mailto:contact@getportal.org/) [us](https://discord.gg/ZXQDuTGcCf){target=_blank} as soon as you are ready. - -Most apps that you install on your Portal need to persist data in some way. There are currently two ways they can do that: a) they request a database on the Postgres instance that is running on the Portal or b) they mount a part of the Portal's filesystem. - - - -Both options isolate the data of each app from those of all other apps and the Portal itself. The database is exclusively used by the app and directories are mounted from an app-exclusive subdirectory. This prevents any sharing of data between apps and ironically mirrors the way SaaS products are working today - one data-silo per application. We find that pattern annoying and don't want it to be repeated on Portal. - -![Directories](screenshot_dirs.png) - -If you look at basically any operating system, you will not find this kind of data-silos. There is a filesystem and application can use it to freely read and write files. This gives the user much more freedom and control, so with Portal we do something similar. - -The feature is called "shared directories" and it is described in detail [in the documentation](/developer_docs/persisting/#shared-directories){target=_blank}. In short, we have added a few directories that are meant to be shared between apps. - -An app can request access to one or more of them and Portal will mount them into the Docker container when starting the app. Changes made by one app are seen by all other apps that have the directory mounted. - -This also allowed us to bring a few new apps to the store that were previously blocked by the missing feature. [Navidrome](https://www.navidrome.org/){target=_blank} is like a selfhosted Spotify for all your music and [Photoprism](https://photoprism.app/){target=_blank} lets you view and organize all the photos and videos that you make. In addition, the good old [Filebrowser](https://filebrowser.org/){target=_blank} can now access all shared directories, so you can use it to view and edit their file structure. - -![Navidrome](screenshot_navidrome.png) -![Photoprism](screenshot_photoprism.png) - -We are planning to release more apps that can take advantage of shared directories. [Tell us](https://feedback.getportal.org/){target=_blank} if you can think of any that you would like to see. diff --git a/docs/blog/posts/2022-06-28_shared_directories/screenshot_dirs.png b/docs/blog/posts/2022-06-28_shared_directories/screenshot_dirs.png deleted file mode 100644 index d7947c3..0000000 Binary files a/docs/blog/posts/2022-06-28_shared_directories/screenshot_dirs.png and /dev/null differ diff --git a/docs/blog/posts/2022-06-28_shared_directories/screenshot_navidrome.png b/docs/blog/posts/2022-06-28_shared_directories/screenshot_navidrome.png deleted file mode 100644 index e721f57..0000000 Binary files a/docs/blog/posts/2022-06-28_shared_directories/screenshot_navidrome.png and /dev/null differ diff --git a/docs/blog/posts/2022-06-28_shared_directories/screenshot_photoprism.png b/docs/blog/posts/2022-06-28_shared_directories/screenshot_photoprism.png deleted file mode 100644 index 2401d30..0000000 Binary files a/docs/blog/posts/2022-06-28_shared_directories/screenshot_photoprism.png and /dev/null differ diff --git a/docs/blog/posts/2022-07-25_backup_your_portal/main.md b/docs/blog/posts/2022-07-25_backup_your_portal/main.md deleted file mode 100644 index fe1454f..0000000 --- a/docs/blog/posts/2022-07-25_backup_your_portal/main.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -draft: false -date: 2022-07-25 -authors: - - max ---- - -# Backup your Portal - -This is a big one! Since from the start, we told everyone that Portal is in an early prototype stage. Your data is not safe, and it could get lost at any time. Technically, this is still the case, but now it is possible to download a full backup of all of your data whenever you want. - - - -![Backup](./screenshot_backup.png) - -Simply open the "More" dropdown (formerly "Settings") and select "Download Backup" and the download will start. - -The backup is a zip archive that contains all the data that makes your Portal unique like its internal database, all the documents or media you uploaded, and the state of all of your apps. Feel free to open and explore the archive. - -⚠️ Careful: inside the archive is also your Portal's unique identity (the technical term is "private key"). Don't share it with anyone, or they could impersonate your Portal! - -## Why is this feature so important? - -### Readiness for productive use - -With the backup feature in place, people can finally use their Portal productively which means for more than just testing and demonstration purposes. They can be sure that their data will not be lost as long as they regularly download their backup. - -And if something happens to their Portal, the backup can be used to create an identical replacement. (Although that process is not automated yet.) - -### Building trust - -We always emphasize that Portal works for the user and only for the user since they are the customer. A part of that is preventing vendor lock-in and let people leave Portal whenever they want and take everything with them. With the backup feature, this is trivially easy. There is no proprietary technology in a zip archive. - -In fact, if you are more tech-savvy, you could spin up the apps that you use on your Portal on your own hardware, mount their state from the Portal's backup and just keep using them locally, just as you left them. So the user is always in full control. - -## Future Work - -The backup feature in its current form is a first shot that works for now. But there is much more to be done. A small example: to prevent people from forgetting to do a backup, we will [add a regular reminder](https://feedback.getportal.org/posts/7/remind-people-regularly-to-download-a-backup){target=_blank}. - -But in the long term, we want [backups to be fully automated](https://feedback.getportal.org/posts/4/automatic-backups){target=_blank}. Every night (or at another interval) they should be pushed to a cloud storage (encrypted, of course), where we keep them safe and allow the user to download them whenever they want. After all, one of Portal's promises is to be fully managed. - -If you have other ideas or wishes for backups or for Portal in general, please participate on our [feedback platform](https://feedback.getportal.org/){target=_blank}. diff --git a/docs/blog/posts/2022-07-25_backup_your_portal/screenshot_backup.png b/docs/blog/posts/2022-07-25_backup_your_portal/screenshot_backup.png deleted file mode 100644 index edb21fe..0000000 Binary files a/docs/blog/posts/2022-07-25_backup_your_portal/screenshot_backup.png and /dev/null differ diff --git a/docs/blog/posts/2022-11-01_peer_2_peer_communication/main.md b/docs/blog/posts/2022-11-01_peer_2_peer_communication/main.md deleted file mode 100644 index 8180ea8..0000000 --- a/docs/blog/posts/2022-11-01_peer_2_peer_communication/main.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -draft: false -date: 2022-11-01 -authors: - - max ---- - -# Peer-2-Peer Communication - -It has been some time since our last update here on Patreon. For one thing, we have been busy with lots of smaller tasks - both technical and regarding business development - that do not really warrant a blog post. Some of them made it into our newsletters though. But another reason for the long silence is the next big feature which was a particularly tricky and time-consuming one: peer-2-peer communication. - - - -From the start, a core part of Portal's vision was the ability to connect Portals in order to form a P2P and decentralized network. Conceptually, this is similar to all social networks which allow you to maintain a list of contacts or friends. But where centralized platforms like Facebook, Twitter, etc. still control all the data and see all connections, Portal makes good on its promise of sovereignty. Your peers (which is the term we chose to use) are stored on your Portal - so your private domain - and Portals talk directly with each other, without a middleman. - -Of course, all communication is end-to-end encrypted and authenticated: a neccessary requirement for privacy. This is made possible by using the [IETF HTTP Message Signatures draft standard](https://datatracker.ietf.org/doc/draft-ietf-httpbis-message-signatures/){target=_blank} which is not often used in normal consumer-facing services where annoying passwords are the norm. However, with Portal's unique features, it is a good fit. Each Portal always has a cryptographic identity (that is where its 6-digit ID is derived from), so it has all the material to sign requests. And since its owner uses the same Portal from all their devices, sharing of private keys (which is a terrible idea anyway) is not needed. It really drives home the point that a Portal is its owners identity on the internet. - -![Peers](./peers_view.png) - -With the new feature in place, an app can now get a list of peers and send http requests to them. The sender Portal adds a signature to the request and the receiver portal verifies it. With each Portal having a valid certificate anyway, this results in an encrypted and mutually authenticated channel. - -Since the feature is brand new, no app uses it yet. It will be the task of app developers to modify their apps for that purpose. - -You can read more technical details in [the documentation](/developer_docs/peering/){target=_blank}. - -## Personal Profile and Welcome Screen - -When managing peers, it is important to have real names associated with them, not just the Portal IDs. That means that a Portal's owner should have a way to enter their name such that it is displayed in their peers' contact list. So we had to implement this feature as a requirement for peering but we also took the opportunity to add a related feature that realizes another important aspect of Portal: a public page/welcome screen. - -We often describe Portal as a user's identity on the internet. Their avatar or representative. With Portal's unique combination of properties this makes sense: it is always on and online and owned and controlled by the user and noone else. A first step toward that goal is giving each Portal a public face. Something to present to people that enter the Portal's address. Until now, you would only see the pairing screen. - -![Profile View](./profile_view.png) - -Now, there is a public page that you can fill with text using Markdown syntax. You can describe yourself or your projects, add links to other online profiles and so on. It can be used to tie your whole internet presence together at a single point. - -![Profile Edit](./profile_edit.png) - -We are looking forward to seeing how people use this feature! As always, for any questions, use our [Discord server](https://discord.gg/ZXQDuTGcCf){target=_blank} and for feature requests or other feedback, here ist the [feedback form](https://feedback.getportal.org/){target=_blank}. diff --git a/docs/blog/posts/2022-11-01_peer_2_peer_communication/peers_view.png b/docs/blog/posts/2022-11-01_peer_2_peer_communication/peers_view.png deleted file mode 100644 index 8174d90..0000000 Binary files a/docs/blog/posts/2022-11-01_peer_2_peer_communication/peers_view.png and /dev/null differ diff --git a/docs/blog/posts/2022-11-01_peer_2_peer_communication/profile_edit.png b/docs/blog/posts/2022-11-01_peer_2_peer_communication/profile_edit.png deleted file mode 100644 index 141bf79..0000000 Binary files a/docs/blog/posts/2022-11-01_peer_2_peer_communication/profile_edit.png and /dev/null differ diff --git a/docs/blog/posts/2022-11-01_peer_2_peer_communication/profile_view.png b/docs/blog/posts/2022-11-01_peer_2_peer_communication/profile_view.png deleted file mode 100644 index 6f33c02..0000000 Binary files a/docs/blog/posts/2022-11-01_peer_2_peer_communication/profile_view.png and /dev/null differ diff --git a/docs/blog/posts/2022-12-08_smart_home_and_iot/installed_apps.png b/docs/blog/posts/2022-12-08_smart_home_and_iot/installed_apps.png deleted file mode 100644 index dd907fc..0000000 Binary files a/docs/blog/posts/2022-12-08_smart_home_and_iot/installed_apps.png and /dev/null differ diff --git a/docs/blog/posts/2022-12-08_smart_home_and_iot/main.md b/docs/blog/posts/2022-12-08_smart_home_and_iot/main.md deleted file mode 100644 index 4c81a2c..0000000 --- a/docs/blog/posts/2022-12-08_smart_home_and_iot/main.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -draft: false -date: 2022-12-08 -authors: - - max ---- - -# Smart Home and IoT - -![MQTT](./mqtt.png) - -Recently, we were made aware of some smart home appliances that use MQTT to talk to a backend in order to publish data or receive commands. This backend can be freely configured, it just needs a MQTT broker. - - - -We were thinking: in theory, with its unique set of features, Portal is a great match for such devices and should be able to play the role of the central hub that manages, monitors, automates, etc. - -Portal is private which is important when you use it to send and store measurements from your home since this data can be intimate. Portal is easy to use, so it lowers the bar for the often already pretty technical process of setting up smart home appliances. And it is accessible from anywhere and from any device which gives you a maximum of freedom while interacting with your smart home. - -So over the last few weeks, we set out to make that possible. We started with a simple first goal: have Portal run a MQTT broker and let users configure it easily. - -## New Apps - -New functions for Portal are of course best implemented as apps that the user installs. We chose Mosquitto as the MQTT broker and Cedalo Management Center as a convenient interface for managing the broker's access control rules. Both apps are now available in the app store. - -![New Apps](./installed_apps.png) - -## Entrypoints Feature for Portal Core - -For Mosquitto to be reachable from the internet via MQTT, it must be possible to map the MQTT port (8883) into the Mosquitto container. Until now, Portal only supported mapping a single http port per app, so there was need for an extension. - -We changed the app.json format such that instead of defining a single port, you can now define multiple entrypoints and set the protocol (and thus the exposed port) for each one. The change is described in the [updated documentation](/developer_docs/app_json/#version-32-to-version-40){target=_blank}. The mosquitto app is the first one to make use of that feature. - -## Guides for Common Tasks - -Even with single-click install apps, the actual setup of your devices is still a process that takes some steps. In particular, managing client credentials and permissions is a task that cannot be skipped while retaining reasonable security. So in order to help people along, we wrote a [short step-by-step guide](/user_guides/smart_home/){target=_blank} that covers installation, setup, and a simple smoke test. - -This is the first of many guides we plan to add to our new User Guides section of the documentation. - -## Conclusion - -This is the first step towards establishing Portal as a valuable part of smart home and IoT. We already planned next steps like enabling more apps that can help you automate, monitor, and control. - -If you experiment with the new feature (or if you don't) we would love to hear your thoughts on them. Every kind of feedback helps us improve. You can [write us](mailto:contact@getportal.org/) or use [our feedback platform](https://feedback.getportal.org/){target=_blank} or chat with us directly on [our Discord server](https://discord.gg/ZXQDuTGcCf){target=_blank}. - -Looking forward to hear from you! diff --git a/docs/blog/posts/2022-12-08_smart_home_and_iot/mqtt.png b/docs/blog/posts/2022-12-08_smart_home_and_iot/mqtt.png deleted file mode 100644 index 4004133..0000000 Binary files a/docs/blog/posts/2022-12-08_smart_home_and_iot/mqtt.png and /dev/null differ diff --git a/docs/blog/posts/2023-09-16_app_integration_overhaul/main.md b/docs/blog/posts/2023-09-16_app_integration_overhaul/main.md deleted file mode 100644 index 1570ad0..0000000 --- a/docs/blog/posts/2023-09-16_app_integration_overhaul/main.md +++ /dev/null @@ -1,231 +0,0 @@ ---- -draft: false -date: 2023-09-16 -authors: - - max ---- - -# App Integration Overhaul - -When developing a piece of software, the best way to make steady and sustainable progress is to work in small and self-contained increments. -Each change should have a clear scope that is easy to reason about, to test, and to roll back if needed. -Side effects should be avoided as much as possible. -With the latest update, I did the opposite of that. - - - -# The Inciting Incident - -I wanted to include Overleaf in our app store. -Overleaf is a web-based LaTeX editor that is very popular among scientists and students -and would make a great addition. -It is especially useful because writing with LaTeX can be painfully technical and Overleaf just takes care of most of the details for you. - -However, it turns out that Overleaf has a few requirements in the form of other containers that need to be running on the same machine, -namely a Redis server and a MongoDB database. -And that is something that Portal just did not support. -Now, in the past, I sometimes worked around this by building a new image for the app that includes and starts all the required processes. -But that can be tricky and error-prone and goes against the idea of Docker containers doing only one thing at a time. - -So it was a good time to tackle a task that has been sitting in the backlog for quite some time: -overhauling the app integration system. - -# Taking it all Apart - -![Mechanic and disassembled machine](./mechanic.jpg){ width="500" } - -Handling apps is one of the primary tasks of Portal and because of that it is deeply integrated into many parts of the Portal core. -Assumptions about how apps work and behave are scattered all over the codebase. -Many of them would no longer be true after the overhaul. -So I had to touch a lot of different modules, pull them apart and put them back together again. - -The rest of this post goes over the major changes - planned, and unplanned, and opportunistic ones - that I made along the way. - -# What is an App? - -## Before - -Before the overhaul, an app was essentially three things: - -1. a single Docker image (hosted somewhere) and -2. the `app.json` file that contained metadata and configuration and -3. an image file that was the app's icon. - -During installation, a large `docker-compose.yml` was updated -that included all the apps that were installed on the Portal and the new app was added to it. -The `app.json` contained the necessary information to do that. - -Over time, it turned out that the format of the `app.json` more and more resembled the format of a `docker-compose.yml` file. -Many values that need to be rendered into the `docker-compose.yml` had to be present in the `app.json` file. -I reinvented the wheel but in a more limited way, most importantly with the one-container-per-app limitation. - -## After - -So it was an obvious choice to just use `docker-compose.yml` files directly. -Each app now has its own `docker-compose.yml` file and Portal just starts it. -Right away, the primary goal was achieved: each app can consist of multiple containers. -As a bonus, lots of configuration can be directly expressed in the `docker-compose.yml` file in a well-known format -and can be thrown out of the `app.json` file. - -However, I could not dispose of the `app.json` file completely. -Some Portal-specific configuration was still needed like access control or lifecycle rules -and these cannot be expressed in a `docker-compose.yml` file. -So I kept it, renamed it to `app_meta.json` (because it is more clear and this was the perfect opportunity) -and cleaned it up. - -While I was at it, I also added an option for apps to define the minimum Portal size they need to run. -Overleaf was the first app to use that because on the smallest Portal size, there is not enough memory to run all the required containers. - -And as a small bonus, I also now publish a json schema file for the `app_meta.json` schema (see [here](../../../developer_docs/app_meta_json.md#schema){target=_blank}). -This makes it much easier to write that file because the schema can be used by an IDE for validation and auto-completion. - -Back to the `docker-compose.yml`, it turned out to be insufficient for apps to provide a static file: -some apps need to be configured during installation. -In particular, many app developers will want to add a Portal's base URL, which is specific to each Portal, to the environment of an app. -So instead of providing the `docker-compose.yml` directly, -the app developer now needs to provide a `docker-compose.yml.template` file, which may contain variables in Jinja2 syntax. -During installation, the `docker-compose.yml` file is generated from that. - -So to sum up, an app now consists of the following four artefacts: - -1. all needed Docker images (hosted somewhere) and -2. the `app_meta.json` containing Portal-specific metadata and configuration and -3. the `docker-compose.yml.template` which is a template for the `docker-compose.yml` file and -4. an image file that is the app's icon. - -The docs contain all the details about the new app format [here](../../../developer_docs/overview.md#the-apps-metadata){target=_blank}. - -And by the way, I of course also had to migrate all existing apps to the new format - or at least those that actually were used. - -# App Store - -## Before - -Changing the app format meant that I also had an opportunity to change the app store backend. -Before, the app store was a GitLab repository that contained the `app.json` files and icons of all apps. -Portal would use the GitLab API to query the app store and download the files. -That seemed like a good idea at the time because it was easy to implement. - -However, there were a few problems with that approach. -First, the GitLab API is not very fast. -After opening the app store, a Portal would take a few seconds querying. -Second, the GitLab API was never meant for hundreds or thousands of clients querying it all the time -which is after all what we are aiming for. -I was afraid that we would hit some rate limits at some point. - -## After - -The obvious solution was to move the app store to a CDN. -The app store is just a bunch of static files after all and CDNs are very good at serving those with low latency and high throughput. - -So now, there is a CI/CD pipeline that builds a few files from the app store repository and pushes them to a CDN on Azure. -The different branches of the app store repository are placed under different paths on the CDN so that testing changes or additions -can be done by switching branches in the Portal UI. -The pipeline also creates a summary file that contains all the apps that are available in the app store -with the metadata that is needed for displaying them in the UI. -(You can check it out [here](https://storageaccountportab0da.blob.core.windows.net/app-store/feature-docker-compose/all_apps/store_metadata.json){target=_blank}.) -That way, only a single request is needed to initially load the app store content (excluding the icons of course). - -With a growing app store, I would also like to add a search feature for apps by name or description or keyword. -I am not sure yet, if, and how that could work with a backend that is just a bunch of static files. -If you have any ideas, please [let me know](mailto:contact@getportal.org/)! - -# App Installation - -## Before - -I describe above how before the overhaul, installed apps were put into a large `docker-compose.yml` file, containing all apps. -The Portal core rendered this file but since it was itself a docker container, it could not directly start it - or so I thought at the time. -Instead, there was a systemd service running on the host that watched the file for changes and executed `docker-compose up` when it changed. -This kinda worked, but it was strange and surprising and not very robust. -In particular, installing multiple apps in rapid succession often lead to errors. - -With the new system, there is now one `docker-compose.yml` file per app, -so I had to redo the starting of apps anyway. - -## After - -What I did not realize at the time and would have saved a lot of work is that -by mounting the docker socket, the Portal core container can control the docker daemon directly. -No need for systemd to call docker commands. -That way, the Portal core container can now issue docker-compose commands to start and stop apps -and do anything else that is needed. - -So now the installation process is as follows: - -1. Portal core downloads the app's files from the app store. -2. Portal core updates its internal database with the app's metadata. -3. Portal core renders the `docker-compose.yml` file from the template. -4. Portal core updates the configuration for the Traefik reverse proxy. - -And when the app receives a request, Portal core simply starts the app's docker-compose file. - -Since the Portal core does everything itself now, there is much more control and visibility of app management, -enabling a few more useful features. - -* If the user installs multiple apps in rapid succession, we can queue the installations and execute them sequentially. -* The UI can show a spinner for an app icon as long as the app is still installing. -* The UI can show a running indicator for an app that is currently running. -* When an app fails to install, the UI can show an error message so that the user is at least informed about the problem. - -Of course the last three items need some kind of push mechanism to update the UI. -And you probably know where this is going. - -# Websockets - -This next section is so obvious, the AI-copilot even suggested the correct heading. - -## Before - -Until now, there was no websocket integration at all. -I just did not see it as a priority and the very few times it was needed, polling was good enough. -Now with the changes to the app installation process, I really wanted to show users what is happening in real time. - -## After - -Fortunately, FastAPI has built-in support and makes websockets really easy to add. -I also took the additional step and integrated websockets with the [Python blinker library](https://blinker.readthedocs.io/en/stable/#){target=_blank}, -which I have been using for internal signals for a while now. -So now there is something of a very lightweight internal event bus using blinker -with a subset of events being additionally published to websockets. - -The frontend which is built with Vue.js reacts to websocket events by publishing them to a global event bus -and by updating the VueX store if needed. -This allows me to use the same list of event names and payload schemas across the whole application. - -# Misc - -These were the large changes, but there were also a few smaller ones that were needed or just made sense to do at the same time. - -* I removed the shared Postgres container that was running on every Portal and that apps could use. - The original plan was to add many more of these shared services but now that each app can spin up its own auxiliary containers, - there really is no need anymore. -* There is one docker-compose file that configures the mandatory containers that make up a Portal which are - the Portal core, the Traefik reverse proxy and a container that statically serves the web frontend. - In this file, the containers were defined with the `:latest` tag which meant that each restart could update them to the latest version. - This was a bit uncontrollable, so I define the exact version now and push a new `docker-compose.yml` file from the backend onto each Portal when I do an update. -* With the new app format, apps can in principle mount any host directory. - I prevent this by policy by not adding apps to the store that do this. - However, it is also an opportunity to give some privileged apps more access to the host. - Like the filebrowser app that can now see all files of all apps. -* Since the overhaul included so many deep changes, existing Portals could not be updated automatically by existing means. - So I added a migration tool that can update a Portal to the new format. - Hopefully, I can use it as a template in case of future deep changes. -* App developers must be informed about the new app system and how to use it. - So I overhauled all the pages of the documentation that were related to the changes (which were almost all of them). - -# Conclusion - -The tasks described above were only the ones that warranted a note on the ever-growing and ever-changing section of todos for this feature. -So all in all, this was a large undertaking, one that took me about two months to complete. -The changes were spread out over 48 commits, the first one being on 2023-05-22 and the last one on 2023-07-18 -and touching 103 files. - -Of course, this is not the best way to do things and I only did it because as a single developer, -there is no coordination overhead. -The benefits of having full control, I guess. - -Anyway, I hope you enjoyed this little insight into the development process of Portal -and are eager to try out the new app system. -Feel free to submit your own apps to the app store ([learn how](../../../developer_docs/overview.md){target=_blank}) -and let us know what you think! diff --git a/docs/blog/posts/2023-09-16_app_integration_overhaul/mechanic.jpg b/docs/blog/posts/2023-09-16_app_integration_overhaul/mechanic.jpg deleted file mode 100644 index a3d889e..0000000 Binary files a/docs/blog/posts/2023-09-16_app_integration_overhaul/mechanic.jpg and /dev/null differ diff --git a/docs/blog/posts/2024-04-07_eating_your_own_dog_food/main.md b/docs/blog/posts/2024-04-07_eating_your_own_dog_food/main.md deleted file mode 100644 index 2efddcf..0000000 --- a/docs/blog/posts/2024-04-07_eating_your_own_dog_food/main.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -draft: false -date: 2024-04-07 -authors: - - max ---- - -# Eating your own dog food - -The Portal ecosystem is more than the software that runs on the Portals themselves, -it also consists of a bunch of administrative functions. -Among others things, they - -* create, assign, upgrade, and remove Portals, -* manage Portal resources, -* monitor Portal health -* accept and process payments, -* receive and process app usage reports for [revenue share](/developer_docs/revenue_share). - -Until now, these tools were accessed using a command line interface, -but with increasing complexity I decided that it is time for something graphical. - -There is the saying "[Eat your own dog food](https://en.wikipedia.org/wiki/Eating_your_own_dog_food){target=_blank}", which means that you should use the products that you create. -So the new graphical interface became a brand-new Portal app. -And it really shows the simplicity Portal can provide if used correctly. - - - -## Portal Controller Backend - -The new app for interacting with the backend services is called "Portal Controller". -It comes with a new backend developed specifically for this app -which, for the start, implements some of the functionality that became important recently. -The portal controller backend is a Python app with a REST API built using [FastAPI](https://fastapi.tiangolo.com/){target=_blank} -and hosted using Azure Container Apps. -This is a departure from the existing backend based on Azure Functions -which - after the honeymoon phase was over - turned out to be too cumbersome to work with. -The aim is to migrate all functionality to the new backend and retire the old one although that may take some time. - -At any time, there is exactly one instance of the backend running, but it has no public web UI. -Instead, it offers a REST API -which is specifically built to be called by the new Portal app. -And here, Portal's features regarding identity and authentication really shine. - -## Portal Controller App - -![Portal Controller App](portal_controller_screenshot.png) - -In general, every Portal app can be divided into two parts: -the backend logic running on each individual Portal and the frontend running in the user's browser. -In the case of the portal controller app, the bulk of the logic is in the frontend. -The backend is very simple. -Its only purpose is to serve the frontend and to redirect calls the frontend makes to the `/api` endpoints to the portal core running on the Portal itself. - -What's up with that redirection? To understand that, let me remind you of a few facts about Portal's concept of identity. - -* Every Portal has a unique identifier, the Portal ID. You can see it as part of the URL and in the top left corner of the Portal UI. -* The Portal ID is connected (i.e. is a hash) to its [private/public key pair](https://en.wikipedia.org/wiki/Public-key_cryptography){target=_blank}. -These are cryptographic keys the Portal can use to prove its identity over the internet. -You can look at my Portal's public key [here](https://c0p3x5.p.getportal.org/core/public/meta/whoareyou){target=_blank}. -* The portal core can make requests to other hosts on the internet and sign those requests with the Portal's private key, which means the receiver can verify that the request was made by that Portal and not by someone else. -* The user's devices (Notebook, Smartphone, etc.) have a pairing with the Portal, which is a [JSON Web Token](https://jwt.io/) that is signed by the Portal's private key. - -
- ![Portal ID in the URL](portal_id_url.png) - ![Portal ID in the UI](portal_id_gui.png) -
The Portal ID in the address bar and the web UI.
-
- -Taken all of that together means that **there is absolutely no need for a Portal user to even think about authentication**. -As you can see in the following diagram, requests from the app UI to the portal controller backend -are authenticated by the normal pairing mechanism on their way to the app, -then the forwarding mentioned above happens, where the request goes to the portal core. -There, it is signed with the Portal's private key and then sent to the portal controller backend. -The backend can now verify the signature and without a doubt identify the Portal that made the request. - -``` mermaid -graph BT - frontend(app UI)-- pairing/JWT -->app(portal controller app); - subgraph Portal - app --> core(portal core); - end - core-- signature auth -->backend(portal controller backend); -``` - -What happens here is a small glimpse of the vision I have for Portal -and why every Portal has this seemingly random ID. -It allows every Portal to cryptographically prove that it is the owner of its ID -without any human interaction. -No passwords, no "Login with _something_", no nothing. - -What I want for the future is that: - -1. Portals can talk _to each other_ in the same way - end-to-end encrypted and authenticated, and -2. that apps make use of this feature to provide a seamless experience for the user. - -## What does the Portal App really do? - -I mentioned that the backend of portal controller app (the one running on each Portal, not the one in Azure) is very simple. -It serves the frontend and forwards requests to the portal core. -In fact, it really does not contain any code at all but is based on a nginx image with a custom configuration which is so small, I can show it here in its entirety: - -```nginx -http { - include /etc/nginx/mime.types; - - server { - listen 80; - server_name localhost; - - location /api { ## (1)! - rewrite ^/api/(.*)$ /internal/call_backend/api/$1 break; - proxy_pass http://portal_core; - } - - location / { ## (2)! - root /usr/share/nginx/html; - index index.html; - } - } -} -``` - -1. This is the part that forwards requests to `/api` to the portal core, where a special endpoint accepts calls, applies the Portal's signature, and forwards them to the backend. -2. This is the part that serves the frontend which is just static HTML, CSS, and JavaScript files created with [Quasar](https://quasar.dev/){target=_blank}. - -What is elegant about this setup is that the frontend can just make calls to `/api` and that way directly interact with the backend on Azure. -It does not have to care about the portal core and authentication at all which reduces the mental load for the frontend developer significantly. - -## Authorization - -Until now, I have only talked about authentication, which is about proving who you are. -Equally important is authorization, which is about what you are allowed to do. -So after the portal controller backend has identified the Portal that made a request, it has to decide if the request is allowed. -How does it do that? - -There are many methods to do authorization, for now I have chosen a simple one, based only on a list of permissions that are directly assigned to Portal IDs. -This is an excerpt from the database entry of my own Portal: - -```json -{ - "hash_id": "c0p3x5wgkav3qfcya7y7lh69kyze03hpfc693gzyzy3apls95nzz8zh24k3wqw9ajcv5vwrqw2a5lw6qy55vy6z4e9dlwhfl3544ls2", - "domain": "c0p3x5.p.getportal.org", - "owner": "Max von Tettenborn", - "permissions": [ - "list_portals", - "read_portal", - "modify_portal", - "delete_portal", - "read_revenue_share", - "modify_revenue_share" - ] -} -``` - -The real entry has more fields, but this is the relevant part. -As you can see, I have the permissions to list, read, modify, and delete Portals and to read and modify the revenue share. -These are all the permissions that the portal controller backend currently knows about. -With more features, more permissions will be added. - -When the time comes to involve more people in the administration of the Portal ecosystem, -I can simply assign the necessary permissions to their Portals. -This instantly enables them to use the portal controller app to do their work. - -Though in the long term, this simple system will likely have to be replaced by something like [RBAC](https://en.wikipedia.org/wiki/Role-based_access_control){target=_blank}. - -## Backends for other Apps - -Of course, the portal controller backend knows all existing Portals from the start, it is the one that creates them. -Other apps that want to offer a backend with the described kind of authentication -would have to create entries for Portals at the time of first contact. -That would be no problem at all, every signed request from a Portal also contains the Portal ID. - -Importantly, this, too, would be totally transparent to the user. -Using a Portal to interact with any kind of backend can always be implemented in a way that the user does not have to think about it. -The backend just always securely recognizes the Portal. - -??? note "Implementing this in your own app" - In the portal controller backend, I used the library [requests-http-signature](https://pypi.org/project/requests-http-signature/){target=_blank} to verify the signatures. - However, it assumes that you already have the public key of the sender, - which in this case, you have to get by calling the Portal's `/core/public/meta/whoareyou` endpoint. - So you would have to do this in your `HTTPSignatureKeyResolver` class. - Also, if you want to write asynchronous code, you would need to make extra steps, because the library does not support it. - If you are struggling, feel free to [ask me for help](mailto:max.von.tettenborn@getportal.org) - I went through those struggles already. - -## Conclusion - -When talking to people about the vision of Portal, I often say that ones Portal should be ones digital twin, or ones representative on the internet. -That only works if it has a consistent identity by which it can be recognized by other online entities - -just like we have our appearance and our name by which we are recognized by other people in the real world. - -With the current system of identity in place, there is no more need for passwords or other methods that require user interaction. -I hope that one day, we will look back and shake our heads in disbelief about the trouble we once had to go through. - -!!! note "" - Can you see the value of seamless authentication? - Would you like to build your own Portal app that takes advantage of that feature? - Discuss at the [Portal Discord](https://discord.gg/ZXQDuTGcCf) or tell me your thoughts via [email](mailto:contact@getportal.org). diff --git a/docs/blog/posts/2024-04-07_eating_your_own_dog_food/portal_controller_screenshot.png b/docs/blog/posts/2024-04-07_eating_your_own_dog_food/portal_controller_screenshot.png deleted file mode 100644 index 4c3a19f..0000000 Binary files a/docs/blog/posts/2024-04-07_eating_your_own_dog_food/portal_controller_screenshot.png and /dev/null differ diff --git a/docs/blog/posts/2024-04-07_eating_your_own_dog_food/portal_id_gui.png b/docs/blog/posts/2024-04-07_eating_your_own_dog_food/portal_id_gui.png deleted file mode 100644 index b0029fb..0000000 Binary files a/docs/blog/posts/2024-04-07_eating_your_own_dog_food/portal_id_gui.png and /dev/null differ diff --git a/docs/blog/posts/2024-04-07_eating_your_own_dog_food/portal_id_url.png b/docs/blog/posts/2024-04-07_eating_your_own_dog_food/portal_id_url.png deleted file mode 100644 index b19fd62..0000000 Binary files a/docs/blog/posts/2024-04-07_eating_your_own_dog_food/portal_id_url.png and /dev/null differ diff --git a/docs/blog/posts/2024-09-03_going_paperless/main.md b/docs/blog/posts/2024-09-03_going_paperless/main.md deleted file mode 100644 index 7befe06..0000000 --- a/docs/blog/posts/2024-09-03_going_paperless/main.md +++ /dev/null @@ -1,160 +0,0 @@ ---- -draft: false -date: 2024-09-10 -authors: - - max ---- - -# Going Paperless with Paperless on Portal - -One of the most useful Portal apps is Paperless. -Especially in countries like germany, where paper still plays a big role in everyday life, -Paperless can help you stay on top of your documents and keep your desk and shelves clean. -After optimizing my setup for the last few weeks, -I have arrived at a point I would consider (almost) ideal. -In fact, it is almost fun to scan and organize documents now. -Let me walk you through it. - -![Digitizing your Documents](title.jpg){ width="500" } - - - -## Why use Paperless? - -The need to manage documents is pretty universal in the modern world (sometimes frustratingly so) and there are many ways to do it. -You could keep the paper in ever-growing binders or boxes. -You could scan them and store them on your computer. -Or put them in a cloud service like Dropbox or Google Drive. -Many people probably do a combination of these things. -But they all have their downsides, like the time it takes to scan and organize documents, the risk of losing them, or the privacy concerns of storing them in the cloud. - -Paperless on Portal is a way to get rid or at least significantly reduce these downsides. -With a proper setup and its smart features, it makes organizing fast and effortless. -Since the documents are kept and processed only on Portal, -they are always available and secure and automatically backed up as well as under your exclusive control (like everything on your Portal). -To my knowledge, there is no other service that offers this combination of features. - -Here, you can also see exactly the three main hallmarks that make up the DNA of Portal itself: simplicity, ubiquity, and ownership. - -## Overview - -In a nutshell, my Paperless workflow looks like this: - -``` mermaid -graph LR - Scanner -- send --> email[E-Mail Account]; - email -- consume/process --> Paperless; - Paperless -- check --> Paperless; - Paperless -- search --> Smartphone; - Paperless -- search --> Laptop; - Paperless -- search --> Tablet; -``` - -1. I scan all incoming documents and trash the originals. -2. The scanner directly sends the scanned documents to Paperless on Portal via a dedicated email address. -3. Paperless automatically processes the documents: it extracts text, categorizes the document (date, type, correspondent), and applies tags. -4. Then, I manually look over the consumed documents in Paperless and correct any mistakes. -5. Once organized, I can search for documents on any device using the Paperless web interface. - -Let's now go into more detail on each of these steps. If you want, you can also [create your own setup](#try-it-yourself) and follow along. - -## Scanning - -![Scanning](scanner.gif) - -For fast and easy scanning, I use a duplex feeder scanner but of course, if you are patient, a flatbed will do the trick. -Mine is the _Brother ADS-1700W_ which is small and light and chews through a stack of paper in no time. -It is a delight to use and I can highly recommend it. - -The scanner connects to my Wi-Fi network and has a feature to send scanned documents directly to an email address. -I have saved this as a shortcut so the scan takes a single tap. -The email address it sends to is a dedicated one that I created just for this purpose. - -!!! note "" - Here, I set up a dedicated email account just for incoming documents. - However, you can also use your existing email account if you prefer and have Paperless only consume mails from a specific sender or with a specific subject. - In the future, we want Portal to host its own mail server so you can even skip this step. - -## Consuming - -This email account is monitored by Paperless. Once a minute, it checks for new mails and -consumes all mail attachments as new documents. -So I can scan a document, and it will be in Paperless within a minute. -I sometimes forward emails with attachments I want to keep to this address, so they end up in Paperless as well. -And if I have a document lying around in the filesystem, I can just upload it using the web interface. - -Connecting Paperless to the email account works in the same way you would connect a mail client like Thunderbird or Outlook. - -![Email setup](paperless_email_setup.png) - -Once the email account is set up, you can create rules that tell Paperless how to process incoming emails. -In this case with a dedicated email account, I only need a single rule that consumes all email attachments and then deletes the email. -A nice trick is to add the `consumed` tag to all documents. -That way, I can easily filter for documents that I have not yet checked. - -![Email rule](paperless_email_rule.png) - -## Organizing - -Paperless is able to extract text from the documents and categorize them on its own based on the content. -After a bit of automated learning, it gets quite good at this so -most of the time, there is no need for me to correct anything. - -Nevertheless, with the help of the `consumed` tag, I still double-check every document manually. -After verifying and correcting any mistakes I remove the "consumed" tag. -This usually takes only a few seconds for each document, so no effort at all. - -Some people might need to get used to not having folders in which to put their documents. -In Paperless, all documents are in a single list instead and the organization is done with their metadata and filters. -This is actually more powerful and flexible. - -If you think about it, folders are just a way to organize documents by category, something that tags can do as well. -So instead of putting a document in a folder called _apartment_, you can just add the tag _apartment_ to it. -In fact, tags are more flexible than folders because a document can have multiple tags -but if you want a document to be in multiple folders, you have to make copies of it. -For example, a tax-deductible bill for your apartment can have the tags _apartment_ and _tax_. - -??? note "What if I need to keep the original?" - There are some especially important documents of which you just have to keep a physical copy. - For these, Paperless has a special feature: the _archive serial number_ (ASN). - The idea is this: you still scan the document and manage it in Paperless, but you also keep the physical copy. - In order to find the copy quickly, you assign it an ASN (in Paperless and on a post-it on the original) - and put all your copies in a binder in order of the ASN. - When searching the original, you first search inside Paperless, look up the ASN, - and use it to find the physical copy by doing a [binary search](https://en.wikipedia.org/wiki/Binary_search) in the binder. - -## Searching - -Thanks to this metadata, searching for documents is easy and powerful. -I can filter the list of documents by date, type, correspondent, or tags. - -For example, when the yearly tax declaration is due, I can filter by type _invoice_ and date _last year_ and have a good first selection of documents to include. -Usually, it turns out that some of the invoices are not relevant, so I refine the filter by removing those _correspondents_ from the list. -In just a few minutes, I have a complete list of all relevant documents. - -![Filtering by date](paperless_date_filter.png) - -Another powerful feature is the fact that I now have all of my documents accessible from any device. -I mentioned this before, but it is worth repeating. -Imagine being at a governments office or somewhere similar, realizing you need a document that is at home. -That just won't happen anymore. - -## Conclusion - -There is no need to digitize all of your documents at once. -About a year ago, I just started with the new documents that came in and that has been very helpful already. -Digitizing old documents is still on my todo list, but it is not urgent. - -![My Paperless Stats](statistics.png){ width="400" } - -The benefits of going paperless are already clear: less clutter, easier organization, better accessibility, -and - most importantly - more free time that is not spent organizing and searching for documents. - -## Try it yourself - -I hope this article has inspired you to give Paperless on Portal a try. -If you want, you can start with a free trial. - -[:sparkles: Create Trial Portal](https://trial.getportal.org/?apps=paperless-ngx){ target=_blank .md-button .md-button--primary } - -And if you have any questions or need help with your setup, feel free to join the [Portal Discord](https://discord.gg/ZXQDuTGcCf){ target=_blank }. diff --git a/docs/blog/posts/2024-09-03_going_paperless/paperless_date_filter.png b/docs/blog/posts/2024-09-03_going_paperless/paperless_date_filter.png deleted file mode 100644 index 9865c3e..0000000 Binary files a/docs/blog/posts/2024-09-03_going_paperless/paperless_date_filter.png and /dev/null differ diff --git a/docs/blog/posts/2024-09-03_going_paperless/paperless_email_rule.png b/docs/blog/posts/2024-09-03_going_paperless/paperless_email_rule.png deleted file mode 100644 index ac88f47..0000000 Binary files a/docs/blog/posts/2024-09-03_going_paperless/paperless_email_rule.png and /dev/null differ diff --git a/docs/blog/posts/2024-09-03_going_paperless/paperless_email_setup.png b/docs/blog/posts/2024-09-03_going_paperless/paperless_email_setup.png deleted file mode 100644 index ea00a1b..0000000 Binary files a/docs/blog/posts/2024-09-03_going_paperless/paperless_email_setup.png and /dev/null differ diff --git a/docs/blog/posts/2024-09-03_going_paperless/scanner.gif b/docs/blog/posts/2024-09-03_going_paperless/scanner.gif deleted file mode 100644 index 49c2d1a..0000000 Binary files a/docs/blog/posts/2024-09-03_going_paperless/scanner.gif and /dev/null differ diff --git a/docs/blog/posts/2024-09-03_going_paperless/statistics.png b/docs/blog/posts/2024-09-03_going_paperless/statistics.png deleted file mode 100644 index 2475769..0000000 Binary files a/docs/blog/posts/2024-09-03_going_paperless/statistics.png and /dev/null differ diff --git a/docs/blog/posts/2024-09-03_going_paperless/title.jpg b/docs/blog/posts/2024-09-03_going_paperless/title.jpg deleted file mode 100644 index 84e2209..0000000 Binary files a/docs/blog/posts/2024-09-03_going_paperless/title.jpg and /dev/null differ diff --git a/docs/blog/posts/2025-04-28_rebrand_and_source_public/Freeshard_logo_construct.png b/docs/blog/posts/2025-04-28_rebrand_and_source_public/Freeshard_logo_construct.png deleted file mode 100644 index 1f1e041..0000000 Binary files a/docs/blog/posts/2025-04-28_rebrand_and_source_public/Freeshard_logo_construct.png and /dev/null differ diff --git a/docs/blog/posts/2025-04-28_rebrand_and_source_public/Freeshard_logo_for_light.png b/docs/blog/posts/2025-04-28_rebrand_and_source_public/Freeshard_logo_for_light.png deleted file mode 100644 index 9cc3273..0000000 Binary files a/docs/blog/posts/2025-04-28_rebrand_and_source_public/Freeshard_logo_for_light.png and /dev/null differ diff --git a/docs/blog/posts/2025-04-28_rebrand_and_source_public/main.md b/docs/blog/posts/2025-04-28_rebrand_and_source_public/main.md deleted file mode 100644 index 950f1af..0000000 --- a/docs/blog/posts/2025-04-28_rebrand_and_source_public/main.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -draft: false -date: 2025-04-29 -authors: - - max ---- - -# Rebrand and Public Source Code - -The brand _Portal_ was always meant to be a placeholder, one that felt increasingly stale lately. -So I did some brainstorming and sketching and came up with a new name and logo. -From now on, Portal will be called: - -![freeshard logo](Freeshard_logo_for_light.png){ width="500" } - - - -## About Portal - -To be totally honest, I came up with the name _Portal_ after playing the Valve game with the same name. -At the time I really had no good name for the project and Portal felt kinda ok: short, easy to remember. -I did not know how serious all this was going to be, so the bar for branding was pretty low. -A collaborator later found the logo on a stock image database. -So it was all a bit low-effort, just to have something. - -Of course the name Portal has its problems. -It is a very common noun, one that is even more commonly used in the IT area. -That makes it hard to find a domain name - getportal.org is unwieldy and the _.org_ TLD doesn't really fit - and search engines won't place it prominently. -It is also weird from a semantic point of view: -A portal is like a door, a gateway, leading to somewhere. -Where does the product _Portal_ lead? Nowhere, really. -It is more like a destination, a thing in itself. - -So all this did not fit and after a long time deprioritizing the rebranding this feeling just grew and grew until I finally put the task on the table. - -## About Freeshard - -Coming up with a new name - a new identity really - still took quite some time. -This was a creative process and such a process cannot be rushed. -So from time to time I revisited the task, tried to come up with other words and meanings, only to drop it again, waiting for the next sprint of inspiration. - -Some of the ideas I wanted to convey were stability/homeliness/intimacy/faithfulness, freedom/independence/self-control, value/uniqueness. -I played around with variations and translations of words like _one, avatar, daemon, deck, hub/center_ and others. -In the end, it became _freeshard_, and I am pretty happy with it. - -Freeshard is actually a known term, albeit only in the bubble of old-school online-games. -When someone who is not the publisher of a game hosts their own game server - with or without the publisher's permission - this is called a freeshard. -So the aspect of self-hosting and individual freedom is right there in the traditional meaning of the word. - -Also, the term _shard_ is reminiscent of a piece of a gemstone, hinting at stability, value, and uniqueness. -However, a shard is only a single piece and many shards make a consistent whole. -This is a good analogy for the project, where the peer-to-peer connections of shards will eventually turn individual tools into a much more useful ecosystem. -In fact, I decided that I'll call the individual instances _shards_. - -So all these thoughts went into the name and make it a good fit. -The cherry on top then was the fact that the freeshard.net domain was free - perfekt. - -Next task: creating... - -## The Logo - -I started sketching and doodling whenever I felt like it. -With the term _shard_ as part of the name, I wanted to make some kind of gem-like faceted structure but also include a certain technologyness. -Here are some of the sketches, the final design evolved surprisingly quickly. -It just kinda made sense. - -![Sketches 1](sketches1.png) -![Sketches 2](sketches2.png) -![Construct](Freeshard_logo_construct.png) - -The simplicity with only two angles for the outlines, the symetry, the "shardiness". -It reminds me of a rocket taking off but the lines can also be seen as wireless transmission signs like in the wifi symbol. -Or you see the central element as a shard and the small elements as its paired devices. -If I sqint real hard, it even reminds me of the Starfleet logo. -All of this was not planned of course, but it is always nice if you can make up origin stories for your logo after the fact. - -## Move to GitHub and Fair Source - -While I was already modifying the project all over the place for the rebranding, it makes sense to tackle other big changes at the same time, right? -No, it does not, in fact it goes against all recommendations for software development, which somehow does not prevent me from [doing it anyway](../2023-09-16_app_integration_overhaul/main.md). - -Your shard is supposed to be your _home on the internet_ (as per the tagline), a place you can trust. -A tiny, unknown startup company offering an online service is not really the best way to build that trust though. -But when you show people the source code of the product, even let them host it themselves, things are different. -So that is the next obvious step for freeshard. - -Unfortunately, for publishing source code _GitHub_ really is the most popular and widely used service in town, by a huge margin. -And Portal was hosted on Git**Lab**, so it had to move. - -That move was surprisingly not as easy as I anticipated. -An org and some repos are created quickly but rewriting the CI/CD pipeline took some more effort -and making the code ready to be published and easy to run for everyone who wants to try it even more so. -Until that point, Portals only existed as their fully managed variant, a much more forgiving context. - -And then there is the license. -I am not sure yet, if I chose the right one using a [functional source license](https://fair.io/licenses/){ target=_blank } -but at least I think I did not shoot myself in the foot with it, and that is something. -Maybe it will change in the future. -As of now, I am glad people can read the source code and host their own shards which will hopefully help build trust -and at the same time, I am the only one who may build a commercial service out of freeshard. - -## Conclusion - -So that was another huge change for the project. -It feels more mature now, although the basic functions are still the same. -I am hoping now for a community to evolve with all its unforeseeable and chaotic and productive side effects. - -You can find the GitHub repo [here](https://github.com/FreeshardBase/freeshard){ target=_blank }. Give it a star if you like. -Try to run it locally, it should only take a few minutes, [here are the steps](https://github.com/FreeshardBase/freeshard?tab=readme-ov-file#localhost){ target=_blank }. -If you run into problems or have an idea for a feature, [open an issue](https://github.com/FreeshardBase/freeshard/issues){ target=_blank }. - -Managed shards are not available yet, but they will come soon. -In the meantime, you can still [try](https://trial.getportal.org/){ target=_blank } or [buy](https://getportal.org/#subscribe){ target=_blank } a managed Portal (which is still essentially the same thing.) - -Hope you have a good time with it, and I am looking forward to your comments at [Discord](https://discord.gg/ZXQDuTGcCf){ target=_blank } or via [email](mailto:contact@freeshard.net). diff --git a/docs/blog/posts/2025-04-28_rebrand_and_source_public/sketches1.png b/docs/blog/posts/2025-04-28_rebrand_and_source_public/sketches1.png deleted file mode 100644 index c00c816..0000000 Binary files a/docs/blog/posts/2025-04-28_rebrand_and_source_public/sketches1.png and /dev/null differ diff --git a/docs/blog/posts/2025-04-28_rebrand_and_source_public/sketches2.png b/docs/blog/posts/2025-04-28_rebrand_and_source_public/sketches2.png deleted file mode 100644 index b92a41e..0000000 Binary files a/docs/blog/posts/2025-04-28_rebrand_and_source_public/sketches2.png and /dev/null differ diff --git a/docs/blog/posts/2026-05-29_unblocking_apps_performance_bottleneck/cover.png b/docs/blog/posts/2026-05-29_unblocking_apps_performance_bottleneck/cover.png deleted file mode 100644 index f700e44..0000000 Binary files a/docs/blog/posts/2026-05-29_unblocking_apps_performance_bottleneck/cover.png and /dev/null differ diff --git a/docs/blog/posts/2026-05-29_unblocking_apps_performance_bottleneck/main.md b/docs/blog/posts/2026-05-29_unblocking_apps_performance_bottleneck/main.md deleted file mode 100644 index 216f120..0000000 --- a/docs/blog/posts/2026-05-29_unblocking_apps_performance_bottleneck/main.md +++ /dev/null @@ -1,70 +0,0 @@ ---- -draft: false -date: 2026-05-29 -authors: - - max ---- - -# Unblocking Apps by fixing a Performance Bottleneck - -![Cover image](cover.png) - -For a while now there was a problem with the Immich app. On the web it did not start most of the time. If it did, it was painfully slow to start and to load images. Curiously, the mobile app did work. I had been meaning to fix that app in particular, but did not get around to it. But a few days ago, a pass over all the apps to upgrade them revealed the actual problem. It was the same problem that plagued other apps and made them slow, for example KitchenOwl. These apps did work, but they were no fun to use. Long waiting times between action and reaction made them just feel sluggish. - - - -## How requests travel inside a Shard - -Every request that hits a shard goes through Traefik first. It is then decided whether that request may pass on to the app or is blocked. All requests from paired devices may pass on as well as requests that target a public endpoint. This is decided by the shard core application. And that means every request gets forwarded by Traefik to that application where the decision needs to happen. It is like a bouncer who has to check your ID for every single sip of your drink, not just once at the door. - -In order to make that decision, the shard core needs to query a few bits of information. What kind of app is targeted? How is its permission model configured? What device is requesting access? This is all in the database, so the database needs to be queried. And to query the database, a database connection is required. This connection is not created each time, but pulled from a pool of connections that are standing by and can be used and then returned to the pool. - -```mermaid -flowchart TD - Browser([Paired browser]) - Traefik[Traefik] - ShardCore[Shard Core
/internal/auth] - DB[(Database)] - App[App
e.g. Immich] - - Browser -->|HTTPS request| Traefik - Traefik -->|forwardAuth| ShardCore - ShardCore -->|find app + identity| DB - ShardCore -->|allow/block| Traefik - Traefik -->|forward| App - App -->|response| Browser -``` - -## The Bottleneck - -As it turned out, each single request pulled two connections from the pool to answer the question of whether it is allowed or must be blocked. Also the pool had only four connections available at a time. This is the default setting and I never questioned it, I did not even consciously see it. But when an app opens 30 or 40 requests on its first start, only the first two can be served right away. All the others are waiting for database connections to be returned and then given out again. A whole bunch of requests block for a long time. The crowd piles up behind the rope while the bouncer works through them two at a time. - -In fact, for apps like Immich or Actual, which open lots of connections during their startup, the long tail of piled-up and waiting requests frequently hit the 45-second browser timeout and the app fails to load at all. - -Debugging that problem was especially tricky because those blocked requests never hit the running application. The browser just gave up on them after they took too long. In the application logs, nothing could be found. There just was no real detectable error happening. It was just a performance degradation that had the effect of an unresponsive application. But none of the observable side effects of real exceptions. - -## The obvious fix is insufficient - -Now, the first instinct would be to increase the database pool size from its default of 4 to something like 20. This would immediately make it possible for many more requests to be served at the same time. However, this just moves the goalposts and apps that make even higher request bursts would still hit a similar problem. Also, getting a connection from a pool, using it to query the database and returning it is not free, there's also a latency involved in that which would still be added to every request. So there must be a better way. - -After taking a look at what is actually queried from the database for each request, the solution became pretty clear. Because the information needed was not something that changed very often. It is the identity of the shard and the requesting device. And it is the metadata and state of the app. Those fields do not change in milliseconds or seconds, but rather in minutes, hours or days. So, they can be cached. But of course, whenever you cache something, you need to think about invalidating the cache which is often described as one of the two hard problems in computer science next to naming things and off-by-one errors. - -## Blinker-Signals - -Quick side note: shard-core uses a simple signal-bus based on the [Blinker](https://blinker.readthedocs.io/en/stable/){ target=_blank } library. A signal is just an object that can be invoked and subscribed to. When it is invoked, it calls all of its subscribers and that way the invoking function does not need to know what those subscribers are. In cases where a signal must be invoked from multiple different locations, this can reduce an n-to-n complexity (each caller calling all consumers) to two one-to-n complexities (n callers to one signal, one signal to n consumers). - -I included the system in shard-core because I found it conceptually elegant at the time, but after working with it, it turned out to not really be needed that much (the n being mostly small) and being its own source of problems and complexity, so I considered for quite a while to remove it and just do the calls directly. - -But to get back to the cache invalidation problem, this is where the signal-bus really came in handy. It allowed me to set up caches that invalidate themselves based on a signal they receive: at every point where the cached data is modified, a signal (like `on_apps_update` or `on_identity_update`) is emitted that notifies consumers of that modification. The cache invalidator is just one of those consumers. So now the mutation sites do not even know about the cache. - -Thanks to the ease of this fix, which was due to the existing signals, I am now leaning again toward keeping them. - -## The result: zero database connections - -With the cache in place, the number of database connections on the hot path of every request is now at zero. The effect on the apps is stunning. They load quickly, they are fun to use. With Immich, I almost cannot scroll fast enough to see the thumbnails loading. - -The patch added 60 lines of code and removed 25. And it transformed a painful experience into a fun one. And that without changing the UI or requiring complicated setup or migration or anything. Really good efficiency. Really good effect for that small of a change. - -My two key takeaways are these. 1. Performance and speed is not a soft target. It is not nice to have. It can fundamentally change the experience. I've seen a talk by Linus Torvalds when he first introduced Git and the audience at Google (using SVN) didn't get it, but he was right. And this fix was a really good reminder. 2. Database connections and queries, even if the database is on the same host, add a significant overhead. If you do too many on a hot path, it will add up and be noticeable. - -The update is already deployed on all existing and future shards. If you have one, maybe revisit apps you found slow before and try them again. If you don't have one, you are invited to a free 24-hour trial [here](https://freeshard.net/en/trial/){ target=_blank }. diff --git a/docs/blog/posts/2026-07-25_app_sleep_fast_wake/cover.png b/docs/blog/posts/2026-07-25_app_sleep_fast_wake/cover.png deleted file mode 100644 index 424953e..0000000 Binary files a/docs/blog/posts/2026-07-25_app_sleep_fast_wake/cover.png and /dev/null differ diff --git a/docs/blog/posts/2026-07-25_app_sleep_fast_wake/main.md b/docs/blog/posts/2026-07-25_app_sleep_fast_wake/main.md deleted file mode 100644 index 176e758..0000000 --- a/docs/blog/posts/2026-07-25_app_sleep_fast_wake/main.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -draft: false -date: 2026-07-25 -authors: - - max ---- - -# Putting apps to sleep, and waking them in under two seconds - -![Cover image](cover.png) - -A shard is a small server. Often a very small one, a VPS with 1 or 2 GB of RAM, and the whole point is that you can install a pile of apps on it and treat them as your own: photos, documents, notes, a recipe box. The trouble is that self-hosted apps are not shy about memory. Immich alone, with its machine-learning workers, will happily take a large bite out of a 2 GB box before you have uploaded a single photo. Install three or four apps like it and the box is full. - - - -## The problem with a personal cloud - -The obvious fix is to stop the apps you are not using and start them again on demand. We already did that. But cold-starting a container stack is slow, often 30 seconds or more, and that half-minute is exactly where the experience falls apart. One of our owners put it plainly: the single biggest daily annoyance was waiting for an app to wake up. It gets worse with Immich specifically, because its Android app polls the server so rarely that it can take up to ten minutes to even notice the instance is back. That is not something we can fix directly, we don't own the app. The only lever we have is on the server: wake faster. - -So the goal became a familiar one with an unfamiliar setting. **Scale-to-zero, the thing you normally only get from big-cloud serverless, but running on a cheap shard with strict per-app isolation.** Idle apps should give their memory back, and waking one should feel instant, not like restarting your computer. - -## The answers that didn't work - -The first instinct is checkpoint/restore: freeze a running process to disk with all its memory, then restore it later exactly where it was. On Linux that means [CRIU](https://criu.org/), or `docker checkpoint` which wraps it. It is a beautiful idea and it has been "experimental" for years. There is no real docker-compose support, and our apps are compose stacks, not single containers. Ruled out. - -The next thought is to switch the whole runtime to Podman, which has friendlier primitives for some of this. But our entire app catalog is built and tested against docker-compose, and migrating the runtime is an ecosystem cost that buys us one feature. Not worth it. - -Then there is [Sablier](https://sablierapp.dev/), a nice middleware that starts and stops containers on demand behind a proxy. We looked hard at it. The problem is that it only starts and stops, and stopping is the slow cold path we are trying to avoid. It also wants to own the wake decision, which would fight the control loop we already run to decide what to demote under memory pressure. Two schedulers arguing over the same containers is a bug waiting to happen. - -What actually worked turned out to be boring in a good way. - -## Freeze, then page out - -Two Linux primitives do the whole job. - -The first is the **[cgroup freezer](https://docs.kernel.org/admin-guide/cgroup-v2.html)**, which you reach through `docker compose pause`. Pausing a container does not stop it. It freezes every process in it in place, instantly, using no CPU. The catch is that a frozen app still holds all its memory, so on its own, pause saves only CPU, not RAM. - -The second primitive fixes that. [cgroup v2](https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files) exposes `memory.reclaim`, which tells the kernel to push a cgroup's anonymous pages out to swap. So the sequence is: freeze the app, then proactively reclaim its memory to swap. The RAM comes back for everything else, and the frozen app sits there costing nothing but some disk. - -Anonymous pages are the memory an app allocates for its own working state, your data in flight, as opposed to file-backed pages that are just cached copies of things already sitting on disk. They're the pages that would otherwise be lost if you stopped the app, and on the apps we care about they're the bulk of the footprint. The target we set is to hand back roughly three-quarters of a running app's RAM this way. - -Waking it is the payoff. There is no cold start, no re-pulling images, no waiting for a database to come up. It is a single `docker compose unpause`, and the app is back exactly as it was, in **under two seconds**. That number is not aspirational; it is asserted by an integration test that pauses and pages a real container stack and then measures the unpause. Compare that to the 30-plus seconds of a full cold start. - -Underneath, apps move through three states rather than two: - -- **Running** — normal, using RAM. -- **Paused + Paged** — frozen, memory mostly pushed to swap, wakes in ms to ~2s. -- **Stopped** — cold, memory fully freed, the slow 30s+ wake. - -An idle app drops from Running to Paused first, the warm state, and only falls to Stopped if it stays idle long enough or the box comes under real pressure. The demotion is driven by [PSI](https://docs.kernel.org/accounting/psi.html), the kernel's pressure-stall information, read from `/proc/pressure/memory`. When memory gets tight, the least-recently-used apps get demoted, and pausing is always preferred over stopping, because pausing is cheap to undo and stopping is not. - -Cold starts don't disappear entirely. An app that stays idle long enough, or one caught when the box is genuinely out of memory, still falls all the way to Stopped and pays the full 30s+ wake the next time you reach for it. And the very first start after you install or update an app is always cold, there's nothing frozen to come back to yet. Sleep just makes the slow path the exception instead of every single wake. - -## Making a cheap VPS survive swap - -Leaning on swap has an obvious risk: swap is slow, and a box that thrashes on wake is worse than one that was just slow to begin with. Two things keep that from happening. - -First, **zswap**. Before a page goes to the swapfile on disk, it goes through a compressed cache held in RAM (zstd compression, decompressing in microseconds). On a small VPS most of the paged-out memory never touches the disk at all; it lives compressed in a slice of RAM and comes back almost for free. Every shard now gets a RAM-sized swapfile plus zswap, because the paging half is useless without somewhere to page into. - -Second, it shipped behind a kill-switch, with telemetry: transition counters, pause and unpause latency percentiles, PSI snapshots, swap usage, all reported so we can watch the real behavior rather than trust the design doc. We've only just started watching, though, so I can't hand you real-world numbers yet, that's the next thing to report. - -## Where this leaves you - -If you own a shard, your idle apps now get out of the way. They give their memory back to whatever you are actually using, and when you come back to one, it is usually there in about the time it takes to notice you clicked. A 2 GB box can hold more of your stuff and still feel responsive, which is the entire promise of a personal cloud that behaves like one product rather than a pile of servers. - -The interesting part, at least to me, is that the answer was not the exotic one. Checkpoint/restore is the flashy idea everyone reaches for; the thing that actually shipped is a freezer, a swapfile, and a compressed cache, primitives that have been in the kernel for years. Sometimes the boring tools win. - -Is there any app you host where wake time drives you up the wall? [Tell me which one](mailto:contact@freeshard.net) and how you use it. And as always, if something isn't behaving the way it should, tell me that too. The idle heuristics are still being tuned, and real usage is what tunes them, just as real bug reports are what fix bugs. diff --git a/docs/index.md b/docs/index.md index ebeb3d1..412fa04 100644 --- a/docs/index.md +++ b/docs/index.md @@ -12,4 +12,4 @@ For a general overview of freeshard and its concepts, see the [Freeshard Overvie If you are a developer and want to know more about developing or adapting applications for freeshard, see the [Freeshard Developer Docs](developer_docs/overview.md). -We also have a [blog](blog/index.md) and some [user guides](user_guides/password_management.md). +We also have a [blog](https://freeshard.net/en/blog/) and some [user guides](user_guides/password_management.md). diff --git a/mkdocs.yml b/mkdocs.yml index 58e3f01..61ad00a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -52,14 +52,11 @@ nav: - 'developer_docs/revenue_share.md' - 'developer_docs/existing_apps.md' # - 'developer_docs/app_template.md' - - Blog: - - 'blog/index.md' - User Guides: - 'user_guides/password_management.md' - 'user_guides/smart_home.md' plugins: - - blog - glightbox extra: