Project Goals

Main goal of the project was to write an app that will automatically handle updates on Gentoo Linux systems and send notifications with update summaries. More specifically, I wanted to:

1. Simplify the update process for beginners, offering a simpler one-click method.
2. Minimize time experienced users spend on routine update tasks, decreasing their workload.
3. Ensure systems remain secure and regularly updated with minimal manual intervention.
4. Keep users informed of the updates and changes.
5. Improve the overall Gentoo Linux user experience.

Progress

Here is a summary of what was done every week with links to my blog posts.

Week 1

Basic system updater is ready. Also prepared a Docker Compose file to run tests in containers. Available functionality:

• update security patches
• update @world
• merge changed configuration files
• restart updated services
• do a post-update clean up
• read elogs
• read news

Links:

• Pull requests: #2, #3, #4
• Docker tests

Week 2

Packaged Python code, created an ebuild and a GitHub Actions workflow that publishes package to PyPI when commit is tagged.

Links:

• Pull requests: #5
• ebuild commit
• GitHub Actions workflow

Week 3

Fixed issue #7 and answered to issue #8 and fixed bug 908308. Added USE flags to manage dependencies. Improve Bash code stability.

Links:

• Issues: #7, #8
• Bugs: 908308
• Pull requests: #6, #9, #10

Week 4

Fixed errors in ebuild, replaced USE flags with optfeature for dependency management. Wrote a blog post to introduce my app and posted it on forums. Fixed a bug in --args flag.

Links:

• Pull requests: #11
• request to fix ebuild
• Blog post and Forum post

Week 5

Received some feedback from forums. Coded much of the parser (--report). Improved container testing environment.

Links:

• Improved dockerfiles

Weeks 6 and 7

Completed parser (--report). Also added disk usage calculation before and after the update. Available functionality:

• If the update was successful, report will show:
  • updated package names
  • package versions in the format “old -> new”
  • USE flags of those packages
  • disk usage before and after the update
• If the emerge pretend has failed, report will show:
  • error type (for now only supports ‘blocked packages’ error)
  • error details (for blocked package it will show problematic packages)

Links:

• Pull requests: #12, #13

Week 8

Add 2 notification methods (--send-reports) – IRC bot and emails via sendgrid.

Links:

• Pull requests: #14, #15

Week 9-10

Improved CLI argument handling. Experimented with different mobile app UI layouts and backend options. Fixed issue #17. Started working on mobile app UI, decided to use Firebase for backend.

Links:

• Pull requests: #16
• Issues: #17

Week 11-12

Completed mobile app (UI + backend). Created a plan to migrate to a custom self-hosted backend based on Django+MongoDB+Nginx in the future. Added --send-reports mobile option to CLI. Available functionality:

• UI
  • Login screen: Anonymous login
  • Reports screen: Receive and view reports send from CLI app.
  • Profile screen: View token, user ID and Sign Out button.
• Backend
  • Create anonymous users (Cloud Functions)
  • Create user tokens (Cloud Functions)
  • Receive tokens in https requests, verify them, and route to users (Cloud Functions)
  • Send push notifications (FCM)
  • Secure database access with Firestore security rules

Link:

• Pull requests: #18
• Mobile app repository

Final week

Added token encryption with Cloud Functions. Packaged mobile app with Github Actions and published to Google Play Store. Recorded a demo video and wrote gentoo_update User Guide that covers both CLI and mobile app.

Links:

• Demo video
• gentoo_update User Guide
• Packaging Github Actions workflow
• Google Play link
• Release page

Project Status

I would say I’m very satisfied with the current state of the project. Almost all tasks were completed from the proposal, and there is a product that can already be used. To summarize, here is a list of deliverables:

1. Source code for gentoo_update CLI app
2. gentoo_update CLI app ebuild in GURU repository
3. gentoo_update CLI app package in PyPi
4. Source code for mobile app
5. Mobile app for Andoid in APK
6. Mobile app for Android in Google Play

Future Improvements

I plan to add a lot more features to both CLI and mobile apps. Full feature lists can be found in readme’s of both repositories:

• CLI app upcoming features
• mobile app upcoming features

Final Thoughts

These 12 weeks felt like a hackathon, where I had to learn new technologies very quickly and create something that works very fast. I faced many challenges and acquired a range of new skills.

Over the course of this project, I coded both Linux CLI applications using Python and Bash, and mobile apps with Flutter and Firebase. To maintain the quality of my work, I tested the code in Docker containers, virtual machines and physical hardware. Additionally, I built and deployed CI/CD pipelines with GitHub Actions to automate packaging. Beyond the technical side, I engaged actively with Gentoo community, utilizing IRC chats and forums. Through these platforms, I addressed and resolved issues on both GitHub and Gentoo Bugs, enriching my understanding and refining my skills.

I also would like to thank my mentor, Andrey Falko, for all his help and support. I wouldn’t have been able to finish this project without his guidance.

In addition, I want to thank Google for providing such a generous opportunity for open source developers to work on bringing forth innovation.

Lastly, I am grateful to Gentoo community for the feedback that’s helped me to improve the project immensely.

Introduction

This article will go through the basic usage of gentoo_update CLI tool and the mobile app.

But before that, here is a demo of this project:

gentoo_update CLI App

Installation

gentoo_update is available in GURU overlay and in PyPI. Generally, installing the program from GURU overlay is the preferred method, but PyPI will always have the most recent version.

Enable GURU and install with emerge:

eselect repository enable guru
emerge --ask app-admin/gentoo_update

Alternatively, install from PyPI with pip:

python -m venv .venv_gentoo_update
source .venv_gentoo_update/bin/activate
python -m pip install gentoo_update

gentoo-update

gentoo-update --update-mode full

export IRC_CHANNEL="#<irc_channel_name>"
export IRC_BOT_NICKNAME="<bot_name>"
export IRC_BOT_PASSWORD="<bot_password>"
gentoo-update --send-report irc

emerge --ask dev-python/sendgrid
export SENDGRID_TO='recipient@email.com'
export SENDGRID_FROM='sender@email.com'
export SENDGRID_API_KEY='SG.****************'
gentoo-update --send-report email

This article is a summary of all the changes made on Automated Gentoo System Updater project during weeks 11 and 12 of GSoC.

Project is hosted on GitHub ( gentoo_update and mobile app).

Progress on Weeks 11 and 12

During last 2 weeks I’ve completed app UI and Firebase backend. Most of the work is done!

I’m not entirely pleased with how the backend works. In Firebase, I ended up using:

• Firestore (security rules defined here)
• Cloud Functions (defined here)
• Cloud Messaging (FCM).

After a user authenticates using anonymous login, a token is automatically registered in Firestore. This token is later used by gentoo_update to send reports. Cloud Functions manage the token’s creation. In fact, all database write operations are handled by Cloud Functions, with users having read-only access to the data they’ve created. Here is how to send the report via token:

export GU_TOKEN="<token ID>"
gentoo-update --send-report mobile

Internally, gentoo-update talks to a Cloud Function. This function checks the token, then saves the report in Firestore for the user to access.

This differs from the original idea, where I didn’t intend to save reports in Firestore. The initial plan was to have the client side listen and let Firebase route report content from the Gentoo system to the app. But this method often missed reports or stored them incorrectly, causing them to vanish from the app. To solve this, I chose to save the reports and tokens, but with encryption.

I’ve came up with a solution to create a custom backend for the app, which users will be to self-host, more about in the Challenges section.

Apart from the web app, I’ve fixed some minor issues in gentoo-update and pushed the latest ebuild version to GURU repository (commit link).

Challenges

While Firebase offers a quick way to set up a backend, it has its drawbacks:

• Not all its best features are free.
• Some of its operations aren’t transparent.
• It doesn’t offer self-hosting.
• Its rate-limiting and security features aren’t as strong as needed. To tackle these concerns, I’m considering a custom backend using this tech stack: Linux + Docker + Python/Django + MongoDB + Nginx.

Here’s a breakdown:

• Django will serve as the backend, handling tasks similar to Cloud Functions.
• MongoDB, a document database, will take Firestore’s place.
• Nginx adds extra capabilities for rate-limiting, load balancing, and security checks.

If necessary, MongoDB can be swapped out for a relational database because the backend will heavily utilize ORM. The same flexibility applies to Nginx.

A highlight of this approach is that everything can be defined in a Docker Compose file, simplifying self-hosting for users.

Plans for Week 13 (final week 🎉)

Here is my plan for the final week of GSoC’2023:

1. Add encryption to the Firestore. I don’t want any user data to be stored in plain text.
2. Improve some UI elements and add a pop-up with commands to copy/paste.
3. Publish mobile app to Playstore.
4. Write a detailed blog post on how to use the whole thing.
5. Writa a post on forums.

This article is a summary of all the changes made on Automated Gentoo System Updater project during weeks 9 and 10 of GSoC.

Project is hosted on GitHub (gentoo_update and mobile app).

Progress on Weeks 9 and 10

I have finalized app architecture, here are the details:

The app’s main functionality is to receive notification from the push server. For each user, it will create a unique API token after authentication (there is an Anonymous option). This token will be used by gentoo_update to send the encrypted report to the mobile device using a push server endpoint. Update reports will be kept only on the mobile device, ensuring privacy.

After much discussion, I decided to implement app’s backend in Firebase. Since GSoC is organized by Google, it seems appropriate to use their products for this project. However, future plans include the possibility of implementing a self-hosted web server, so that instead of authentication user will just enter server public IP and port.

Example usage will be something like:

1. Download the app and sign-in.
2. App will generate a token, 1 token per 1 account.
3. Save the token into an environmental variable on Gentoo Linux.
4. Run gentoo_update --send-report mobile
5. Wait until notification arrives on the mobile app.

I have also made some progress on the app’s code. I’ve decided to host it in another repository because it doesn’t require direct access to gentoo_update, and this way it will be easier to manage versions and set up CI/CD.

Splitting tasks for the app into UI and Backend categories was not very efficient in practice, since two are very closely related. Here is what I have done so far:

• Create an app layout
• Set up Firebase backend for the app
• Set up database structure for storing tokens
• Configure anonymous authentication
• UI elements for everything above

Challenges

I’m finding it somewhat challenging to get used to Flutter and design an modern-looking app. My comfort zone lies more in coding backend and automation tasks rather than focusing on the intricacies of UI components. Despite these challenges, I am 60% sure that in end app will look half-decent.

Plans for Week 11

After week 11 I plan to have a mechanism to deliver update reports from a Gentoo Linux machine.