Hotspot Companion App

[Popolechien]

⬇️

[rgaudin]

Main Objective

A Mobile App to synchronize data between a running Hotspot and an Internet-reachable Kiwix Service.

A Kiwix Hotspot is a raspberry-pi device that's only connected to a power source. It runs our software stack to create a WiFi Hotspot that provides access to embedded content via Kiwix-serve and other tools.

We want to be able to collect arbitrary data from Hotspots to a Cloud service (usage metrics at the moment) and also be able to send back data from the Cloud service to the Hotspots.

Hotspots are deployed to replace Internet connectivity where it's missing or expensive. Most of them are offline, always.

Mobile phones are easy to find in most places and it's reasonable to assume that an Hotspot Admin would have access to one.

We thus want to develop a mobile App that serves as an asynchronous proxy between multiple Hotspots and a Cloud service.

The phone will connect to the Hotspot WiFi network and exchange data with the Hotspot. The phone will connect to the Internet via WiFi or mobile data and exchange data with the Online service. These two phases will repeat over and over and we do not know which one will happen first (user might first connect to Online service or first connect to one or many Hotspot). Time and physical location between the various connections is varying (could be minutes to days or months, could be all in the same place or in various locations).

Important

This is a high-level Project Description. This will be turned into actionable tickets once we have validated a proposal and been attributed a GSoC student.

Feel free to question all choices if you think there is a better/cheaper/more-efficient way to do. If you do though, justify yourself so it's criytal clear that you respect the general requirements (main objective) and that your alternative is beneficial. Keep in mind that while this will be developped during a GSoC, we will have to maintain it for a long time so maintainability is key.

Unclear proposals will be discarded. We need to be sure you've understood the requirement.

Target Users

App will be used solely by Hotspot Managers: a Hotspot manager is a person responsible for one or multiple Kiwix Hotspot deployment.

The manager hold the Admin credentials to said Hotspot(s). Admin credentials are a username and password that are required to access admin functions of the Hotspot(s). Each Hotspot could have different credentials, even if they are all managed by the same Admin. Admin however uses usually the same credentials on all their Hotspot.

Admin has access to a mobile device (Android or iOS). Admin has temporary access to either an Internet connection on this mobile device (Wifi or mobile data) or to the Hotspot Wifi.

User Workflow

Initially

• Installs the App on its phone/tablet (when connected to the Internet, or directly from an APK fetched somewhere)

Wants to extract data from a Hotspot

• Connect to Hotspot WiFi with the mobile device settings (Hotspot companion is not used in this phase)
• Launch App (or reopen it if it was only suspended)
• Create a new Hotspot profile as there should be none on fresh install
  • IP is configurable with default to 192.168.2.1
  • Hotspot credentials are set (reusing credentials from another Hotspot profile or default credentials is a nice-to-have feature)
• Select Hotspot tab
  • auto selecting profile because there's just one profile
  • otherwise profile must be selected
• Displays:
  • List of Data Packages (with size) available on Hotspot
  • List of Data Packages on phone that can be sent to this Hotspot
    • either data for this specific Hotspot or for any Hotspot
    • none at this stage (fresh install)
• Ticks the ones he wants to sync (transfer size is shown)
• Hits Sync to transfer selected Packages to/from phone. Progress bar shows.

➡️ at this point, communication with Hotspot is over

Wants to upload data to Cloud

• Gets an internet connection (at any future point in time and location)
  • directly if Hotspot is connected to Internet
  • by enabling mobile data or by connecting to a different WiFi network with Internet connectivity
• Select Cloud tab
• Enters Cloud credentials (first time only)
• List all Hotspot Data Package and size (from one or multiple hotspot) stored on the phone which have not yet been uploaded to the Cloud.
• List all data that can be downloaded into phone for one or all profiles's Hotspots (with size)
• Tick the one he wants (everything's ticked by default) Total size is shown
• Hits Sync to Upload to Cloud and Download to phone

➡️ at this point, communication with Cloud is over.

Features & Principles

• App is a temporary store for data that has to be moved from/to Cloud service from/to Hotspots
• Data is identified as:
  • from a specific Hotspot
  • for a specific Hotspot
  • for all Hotspot (broadcast)
• Data is dated (date of collection from Hostpot or from Cloud)
• Data is a single file, agnostic of any format (could be a zip, tar, zim ...)
• Total volume of data stored in the phone at a given point in time could be significant (think tens of gigabyte)
• User can manage multiple Hotspots. Those are recorded as Profiles (CRUD)
  • Hotspot Unique identifier
  • Hotspot name (chosen by user, defaulting to Hotspot name)
  • Hotspot credentials
• User has a unique credential (username and password) to the Cloud Service
• User has credentials (username and password) to each Hotspot he managed
• Encryption of sensitive data (credentials) is a nice-to-have feature
• User must be able to visualize what's in the mobile device:
  • Kinds of data for/from each Hotspot, with respective and total size
  • Ability to discard them should he need to reclaim space
• All transfers are logged.
• App works either in Hotspot Context or Cloud Context ; switching between them should however not need an application restart
• Last action for each kind on each end is dated and used by APIs to filter:
  • ex: App says to Cloud Service: I downloaded Kind A from you for Hotspot XYZ on YYYY-MM-DD and that data have been sent to Hotspot on YYYY-MM-DD.
• Connection to Hotspot is done via WiFi only
• App receive arbitrary bytes data from API. Apart from metadata, it doesn't care what the data is.
• Cloud service URL is configurable in settings
• Transfers are manually triggered and a summary is displayed before, with Data Sizes: communication with Cloud service might be on a per-per-mb data connection
• A Data Package is what is exposed in the API. It's a Kind of data. Ex: Metrics.
• When retrieved, a Data Package might be composed of many small packets.
• App should allow specifying where to store its data so devices with limited internal storage but external media (sdcard) can choose to store it there.

Techno

Mobile App Framework

• Our mobile experience is limited
• We need to support both Android and iOS
• Maintainability is key
• We are reluctant to use Hybrid frameworks like ionic due to the painful dependencies maintenance it requires. We want to be able to update the code after 6 months and rebuild the app without troubles.

We've identified Flutter as good candidate:

• Established and supported (created by Google)
• Single codebase for both platforms
• Real native apps as output
• Adaptable UI for each platform

Note that Flutter generate an XCode project and builds with it so a macOS machine is required to build the iOS app.
We don't require candidates to have a macOS (you'll test with android version then) but please confirm in your proposal that you have access to an android or iOS device for tests.

Data transfer

We believe that transferring data via multiple, small HTTP requests would be a good choice:

• very reliable
• client support is good, everywhere
• observable (progress)

Scope of Work

The GSoC project is limited to the mobile App. The expected final deliverables is a repository containing:

• App code matching requirements
• Github CI to ensure app builds properly
• Github CD:
  • nightly builds
  • APK release
  • Testflight release
• All code will be added through reviewed Pull requests, following our contribution guidelines.

Your proposal must include a mockup of the App's User Interface.

You will be provided with the API for both Hotspot and Cloud Service (probably similar) soon (end of March).

[AyushGupta11442]

hello @rgaudin @kelson42 @benoit74 @Popolechien

My name is Ayush Gupta, and I am currently pursuing a Bachelor of Technology degree from SRM University.
I bring with me a strong background in Flutter development and Dart, with nearly three years of experience. Additionally, I am proficient in React, JavaScript, and have knowledge of machine learning concepts.
Given my expertise and enthusiasm for this project, I am eager to contribute and make a meaningful impact. If you have any specific requirements or tasks in mind, I would be grateful for the opportunity to discuss them further.

[Popolechien]

Hi @AyushGupta11442 as you can see the project isn't entirely scoped yet. I'd suggest you poke around the repo (start with good first issues here and on /kiwix or /openzim as we won't consider project proposals from people who haven't submitted a few PRs beforehand.

[aryanA101a]

Gets an internet connection (at any future point in time and location)

Do we have to bridge the internet connection of the mobile device to the raspberry pi to facilitate internet to other connected devices?

Requests online credentials and metadata for profiles that have not been synchronized.

Please provide some context on this point. What are profiles here and for what they are used?

[rgaudin]

Do we have to bridge the internet connection of the mobile device to the raspberry pi to facilitate internet to other connected devices?

No. The app will just connect to the Pi via HTTP on the Hotspot WiFi

Please provide some context on this point. What are profiles here and for what they are used?

The App will allow one person to Sync data to/from multiple Pi with the Online service. It's not clear at this stage if the App user will have a single set of credentials and be able to attach to multiple Pi for instance.
We need a way to make sure that:

• data is uploaded for a particular Pi
• person uploading data for that Pi has the right to do so

[rgaudin]

All, thank you for your comments ; I've updated the project description. It should now be somewhat clear.
We'll add the API by the end of the month so you know exactly what you'll be dealing with but it's gonna be simple: list packages, download them, mark them downloaded, etc.

[aryanA101a]

What's the need to store Hostpot credentials in the user profile?

[rgaudin]

What's the need to store Hostpot credentials in the user profile?

No User profile is mentioned. We want to create a Profile (naming might change) for each Hotspot the user has access to. Because hotspot API require credentials (username, password), storing them on the phone means it will be typed only once and then remembered

[aryanA101a]

What's the need to store Hostpot credentials in the user profile?

No User profile is mentioned. We want to create a Profile (naming might change) for each Hotspot the user has access to. Because hotspot API require credentials (username, password), storing them on the phone means it will be typed only once and then remembered

From Hostpot credentials I inferred the ssid and password of the wifi ap.
Okay, so you meant existing admin credentials to the hotspot.

[rgaudin]

We don't want to manage the WiFi connection from the App. They will connect to the network first then use the app.

We want the app to be as dumb and limited as possible ; to reduce maintenance and because an app in the wild is harder to update.

[aryanA101a]

List of its Data Packages (with size) available on Hotspot

Whats the scope of the data packages we are looking at? Location wise?
Is it limited to the data in the /data/contents/zims.

List of Data Packages on phone that can be sent to this Hotspot
either data for this hotspot or for any Hotspot

I think this requires an asyncrounous .zim file specific search on the internal storage of the phone plus caching of results.

[rgaudin]

@aryanA101a please read the description carefully. You should not care at all about how stuff are stored on the Pi. Your app will talk to an HTTP API to get a list of items and you'll retrieve content for those items through this API.

I think this requires an asyncrounous .zim file specific search on the internal storage of the phone plus caching of results.

I don't understand this sentence. The app won't be looking at stuff on the phone outside its working directory.

[benoit74]

I've also updated the first comment descriptions with:

• More details about Hotspot credentials
• More details about Wifi connection to the Hotspot
• Data is a single file, agnostic of any format (could be a zip, tar, zim ...)
• Total volume of data stored in the phone at a given point in time could be significant (think tens of gigabyte)
• Encryption of sensitive data (credentials) on phone storage is a nice-to-have feature
• Many small fixes

[aryanA101a]

@aryanA101a please read the description carefully. You should not care at all about how stuff are stored on the Pi. Your app will talk to an HTTP API to get a list of items and you'll retrieve content for those items through this API.

I think this requires an asyncrounous .zim file specific search on the internal storage of the phone plus caching of results.

I don't understand this sentence. The app won't be looking at stuff on the phone outside its working directory.

I assumed an extended scope when looking at the packages on the phone, that's why I was trying to clarify.

[Anirban-Majumder]

Hello @rgaudin @kelson42 @benoit74 @Popolechien Just to clarify, considering the way the hotspot is build, which type of API—SOAP or REST—will the app talk to?

[rgaudin]

HTTP REST

[aryanA101a]

Is unit testing in the scope of this project?

[rgaudin]

Up to you

[technosrijan]

@rgaudin Sir, I wanted some clarification regarding whether a GSoC candidate has been selected for this in 2024, as I checked their list of contributors website and couldn't find this project over there.

[kelson42]

@technosrijan I confirm there is no GSoC student working on this issue currently.