GSoC: Settings API and Basic Inward Facing Testing API

[Ocupe]

!!! Proofreading in Progress. The corrected version will follow later. I thought feedback is more important than correct spelling. !!!

GSoC Proposal - Toga Settings API and Basic Inward Facing Testing API

Introduction

Programs and settings are a inseparably bound with each other. Every professional application gives users the freedom to change default values to there needs and liking. Toga should realise this need and provide a strong and compelling solution to create beautiful and platform native settings.

The Toga project is completely untested at the moment. As a first step to tackle the not trivial problem, that cross platform testing is, I want to build up a inward facing testing system to test toga core and it's interface.

Project Goals

In short, the main goal of this Google Summer of Code project is to create a strong and compelling Settings API in toga core that is then also implemented in toga.cocoa and toga.iOS.

Implementation

I'm dividing the Settings API implementation into two stages. Conceptualising and Implementing.

Conceptualising the Settings API

Because Toga aims to be platform independent, what also includes it has to be device independent (pc, mobile), we have to create a Settings API that considers for all of that. In #90 I go into great detail about the main challenge of creating a Settings API that works on mobile as well as on Desktop. Some issues to resolve are.

• What are the minimum properties that a settings widget needs? Find common denominator between platforms.
• Translation of widgets that only exist on mobile or desktop
• At what point in time are the settings created? During runtime or before compile time (like iOS requires)?
• Setting and getting of settings values.

How is the end user going to experience the Settings API?

On key feature of Toga is its use of native widgets that makes the user feel at 'home' on his/her platform machine. The Settings API should not be any different. Users are going to find and open their preferences on the same locations that are normally used for settings. (File > Preferences on Mac, Settings App on iOS ...). The Settings API is also going to provide functionality to save settings in a platform nativ fashion (.plist files on osX/iOS, .xml for Android...).

How is the developer going to use the Settings API?

Another big challenge is to design the Settings API to be easy and a pleasure to use for whom which are actually going to use it - the developers.
Some aspects that I have in mind are:

• Make it a simple and a streamlined process to create settings though the use of context managers (with).
• Give the developer a elegant way to set and get settings values and the possibility to define a function that is called on value change.
• Make it easy to save settings to platform native places.
• Give the possibility for application settings as well as project settings.
• Give a simple solution for settings migration to support versioning.

Under the Hood - a glimps of the implementation

To give a better understanding of how I envision the implementation I give some use cases.

Standard way to use Settings API

import toga

# Instantiating the settings class takes care of creating the settings window/view.
# All the platform specific bootstrapping should be taken care of and set to native defaults.
settings = toga.Settings()

# A the settings group should have a label and the possibility to add a icon to it.
group = toga.SettingsGroup('General'. icon='icon.png')

# creating setting item
switch = Switch(label='Show line numbers')

# adding setting item to the setting group
group.add(switch)

# adding group to settings
settings.add(group)

# add settings to app
app.settings = settings

# you should be able to open the settings window/view by just calling the open() function.
app.settings.open()

Elegant way of using the Settings API:

with app.Settings:
    with toga.SettingsGroup('General'. icon='icon.png'):
        # add all settings items for this group
        toga.Switch(label='Show line numbers')
        toga.Slider(label='Nr of lines to show')
        toga.Switch(label='...', ...)
        ...
    with toga.SettingsGroup(...):
        toga.Switch(label='...', ...)
        ...
        
app.settings.open()

How can other toga contributors integrate the Settings API on their platforms.

I'm aware that there is a universal outside the Apple galaxy and we need to support other platforms as well. I picked macOS and iOS just because I'm comfortable developing for those platforms. I already included Android into my considerations how the core Settings API should look like to not exclude any platform or even make it hart for them to implement the developed Settings API.

As I mentioned in #90 I want a clean Interface between toga.core and the respective platform integration of the Settings API. This interface converts the defined settings in code to a normalised form. That form is then passed to the platform specific integration of the Toga Settings API integration. This is best explained with a simplified example.

# defining the settings
settings = toga.Settings(version='1.0.0')
switch = toga.SettingsItem('switch', label='My Switch', default=True)
slider = toga.SettingsItem('slider', label='My Slider', default=5, min=0, max=10)
group = toga.SettingsGroup('My Settings', [switch, slider])
settings.add_group(group)

After the definition, the platform specific integration of the Toga Settings API, calls the get_normal_form() function to receive the normalised form of the before defined settings, which looks like this:

# When printing the normalised form we end up whit something like this.
pprint(settings.get_normal_form())
{'settings': 
    {
    'version': '1.0.0',
    'groups': 
        [{'group':
            {
            'title': 'My Settings',
            'items':
                [{  
                    'default': True,
                    'key': 'my_switch',
                    'label': 'My Switch',
                    'type': 'switch'},
                {   
                    'default': 5,
                    'key': 'my_slider',
                    'label': 'My Slider',
                    'max': 10,
                    'min': 0,
                    'type': 'slider'
                }]
            }
        }]
    }
}

This agreed upon normalised form than acts as the base for all the system specific integrations. For example, a Android developer can now translate this dictionary to a suitable and platform native representation for a settings view.

The normalised from has the nice side effect that we can easily save it as a platform independent settings file (For example as a xml or json file). This allows also for settings syncing over cloud solutions.

Timeline

>>> GSOC.community_bonding('May 4 - 30, 2017')

• Dive deep into existing native Linux, Windows and Android integration for their settings APIs to find a common denominator and don't exclude any platform though an avoidable design mistake.
• further development of the normal_form
• discussions with the community
• research about how to set up a expandable and good structured testing system for a project like toga

>>> GSOC.week(1, 'May 30 - June 2')

• Setting Up a basic inward facing test system for toga core.
• Setting up a test structure for the upcomming Settings API core classes.
  • toga.src.core.Settings
  • toga.src.core.SettingsGroup
  • toga.src.core.SettingsItem

>>> GSOC.week(2, 'June 5 - 9')

• Implementing class SettingsItem with
  • switch
  • text field
  • single value pick
  • mule value pick
  • slider
• translating SettingsItem to normal_form

>>> GSOC.week(3, 'June 12 - 16')

• Implementing class SettingsGroup
  • Group icon
  • tranlate to normal_form
  • functionality to add SettingsItems

>>> GSOC.week(4, 'June 19 - 23')

• Implement class Settings()
  • how to distinguish between project and application settings
  • tranlate to normal_form
  • show() to open settings window/view
  • add Settings group

>>> GSOC.week(5, 'June 26 - 30')

• Implement
  • Settings extraction
  • save platform independent settings format
  • load settings from file

>>> GSOC.week(6, 'July 3 - 7')

• Implement
  • versioning and migration

>>> GSOC.week(7, 'July 10 - 14')

• Prepare and modify macOS widgets
  • enable widgets to have an on_change callback
• Integrate missing widgets
  • slider
  • switch
  • multi-value element

>>> GSOC.week(8, 'July 17 - 21')

• Implement Settings API for macOS
  • translate normal_form into platform native widgets

>>> GSOC.week(9, 'July 24 - 28')

• Prepare and modify iOS widgets
  • enable widgets to have an on_change callback
• Integrate missing widgets
  • slider
  • switch
  • multi-value element

>>> GSOC.week(10, 'July 31 - August 14')

• Implement Settings API for iOS
  • translate normal_form into iOS Settings Bundle

>>> GSOC.week(11, 'August 7 - 11')

• Implement iOS settings bundle creation into briefcase
  • iOS Settings Bundle has to be created before app compile time

>>> GSOC.week(12, 'August 14 - 2')

Because the iOS documentation recommends a custom UI for frequently changed preferences instead of the settings bundle, I'm going to implement a solution that gives the developer the possibility to deploy their settings to a normal table view as well. In this form the settings view is just another view in the app and not part of the iOS Settings app.

• Implement iOS Settings API stage two
  • translate normal_form into platform native widgets

About me

Hey Bees, thanks for reading my proposal. I'm Jonas and I studied Digital Media. Programming is not a requirement for my masters programs. This should show you that I don't program because I have to but because I love to. Wait, am I then qualified you ask? Even though I'm mostly self taught I had my fair share of university programming courses.

I tried to keep the proposal short, if a more detailed explanation is desired I am happy to expand on any subject.

I look forward to feedback and comments.

Academic History

University of Applied Sciences Düsseldorf

Acquired my Bachelors Degree in Media Engineering. First contact with C and Java and the beginning of my interest for programming.

Drexel University Philadelphia

Because of the performance during my bachelors program I was able to get funding though the Fulbright Foundation. This funding enabled me to study for one year at Drexel University in Philadelphia. I was enrolled in the Digital Media masters program, which was focused on different domains in the field of Digital Media (Game Design and Programming, Interactive Web-development, 3D-Animation and Rendering and Media Theory).

University of the Arts Bremen & University of Bremen

At the moment I am completing my masters degree in Digital Media. My studies here are a joint program of two universities. The University of Bremen, which is responsible for the informatics part of the program and the University of Arts which is dedicated to the media theory and artistic part of the program.

[freakboy3742]

This is pretty good for a first pass (and the spelling isn't even that bad! :-)

The design for the core functionality has already been vetted as part of #90, so you've got a really good head start. There's only two areas that I think need a bit more clarification, and they're the two areas that aren't really covered by #90:

• You make reference to testing, but you don't really elaborate what you've got in mind. We've had some preliminary discussions, but scratching out some broad plans (either in this proposal, or in a separate ticket like Settings API for Toga #90), would be desirable.

• Week 6, you make a reference to versioning and migration. This is definitely needed; but migration was also a project that took many years to add to Django. The task we're faced with is much simpler than Django's needs - but it's also something that hasn't been discussed previously.

[Ocupe]

@freakboy3742 I created #119 to clarify what I had in mind for the testing part. I only consider the inward facing section of #119 as part of this GSoC proposal. Both would be to big in scale I think.

[Ocupe]

@freakboy3742 about versioning and migration.
Maybe I underestimate the task at hand but I was thinking about something along these lines.

Example

v1 and v2are simplified versions of the Settings API. If I migrate from v1 to v2 I would check if something has changed and if so, I would perform the following actions.

v1	v2	Action
A: True	A: True	None
B: (1, 2)		Remove
C: 'nice'	C: 'very nice'	Update
	D: 'new'	Add

I have never done something like this before. Do I miss some key challenge/problem?

[freakboy3742]

Possibly not; migrations for Django required a whole lot of interaction with transactions, database backends, inconsistent syntax, plus dealing with collaboration between developers working in parallel. In retrospect, I guess migration in the settings context is literally just a function that takes v1 format and outputs v2 format.

[freakboy3742]

This proposal was accepted for the 2017 GSoC. Project progress updates will be posted here by @Ocupe.

[Ocupe]

[Logbook 1]

Google Summer of Code has started! This is a quick recap of what I did in week 1 and a preview what I'm planing to do this week.

In the weeks preceding the start of GSoC I tried to figure out what a test setup for toga could look like. With the help of the community, especially @phildini, @dgelessus, I came to the conclusion that toga, and even toga.core are really hard to test. The main problem are platform dependencies that are intertwined with toga.core interfaces.
The main work of week 1 was to come up with ideas how to separate and restructure toga. This is unknown territory for me and I'm thankful for all the input and help from @freakboy3742.

We are already in the mid of week 2 and I'm currently working on restructuring toga to make it testable. If you're interested in the process you can watch my hacking around here (Waring no docs or descriptions at the moment).

I hope we have a simple implementation of what a restructured version of toga could look like by the end of the week.

If you have questions or comments please feel free to contact me on Gitter or post a comment here.

[Ocupe]

[Logbook 2]

This weeks main goal is to validate the new restructured form of Toga. The key points to keep in mind are:

• Does the restructuring make Toga testable?
• Do we create new problems with the new design?
• Is it worth all the work/refactoring?

If the validations outcome it positive we can work on refactoring the rest of Toga.

To change the design of a program like Toga you really have to understand the inner workings of it. In the last weeks I learned a lot about Toga and the processes that make it work. @freakboy3742 suggested to document my learning experience and for others to come. Hence, as I go along I try to write down how Toga works internally. On top of that I try to produce some info graphics about the structure of Toga. Drawing the program structure on paper always helped me, so I thing others could also profit from it.

As a last point I try to introduce static typing and MyPy into Toga. I think Toga could gain some confidence and make itself easier to understand if we have at least some static typing in Toga.core.

If you have questions or comments please feel free to contact me on Gitter or post a comment here.

[Ocupe]

[Logbook 3]

Recap

Last week I finished the proof of concept for the restructured version of Toga. The proof of concept consists of a working Toga-Cocoa and iOS app. Supported widgets are box and button.
The current status is hosed here.

Static Typing

I did some tests with static typing and MyPy on this branch of toga. After consolidating with @freakboy3742 we decided to put is on hold for the moment. However, it is still possible that static typing is introduced later to the project.

Preview

This week I will move the, currently separated, ocupe/restruc_toga repo into a pull request of the pybee/toga repo. This allows us to start the process of making a Toga 0.3 release with the new restructured and tested version of Toga.

After creating the PR, I will, step by step, add more widgets and tests for the Toga-Cocoa and iOS backends.

[Ocupe]

[Logbook 4]

Recap

A lot of code has accumulated in the newly created PR. I think this justifies the name of “The Big Restructure of Toga [WIP]“.
Within this PR I have added the previously separated restructure of Toga as well as some prove of concepts for a test framework of Toga.

Testing Toga

As a reminder, the main reason for this endeavour was to make Toga more testable. Until now, the first tests look promising.

Implementation Checks

The basic idea of the implementation check is to verified if a Toga backend satisfies the requirements in form of classes, methods etc.

The biggest problem is the fact that most of the backends are platform specific. For example, the toga-iOS backend requires a iOS device or at least a simulator to run and even to be imported without an exception. This hinders us for just importing the backend and running tests against it. As a way around this we introduced the ast module. With the ast module we can parse the actual .py file and get a representation of the python code in form of an abstract model tree.

From the tree we can now extract all kind of structural information about the python file. At the moment I just extract what classes and class methods are in the file. With this informations I check if the requirements are met.

The requirements are defined in a newly created dummy backend called toga-dummy. At the moment it is hollow with now functionality, just structure. We extract the required class structure form the dummy backend in the same way we test it, with the help of the ast module.

With the requirements at one hand and the backend representation on the other we can now perform some unit tests and check if all the required classes and methods are implemented.

This is still very basic but we can build on it. We also see the implementation tests as one part of the bigger testing puzzle that should eventually cover all of Toga.

[danyeaw]

@Ocupe Any reason to keep this GSoC proposal still open?

[Ocupe]

@danyeaw I don't see a reason. Let's clean up :)