Visual Phenomics | DataFrame

Import data output from Visual Phenomics into a convenient DataFrame for subsequent analysis in Python.

Installation

Install using pip in the terminal. If you are using Anaconda, make sure, you are using the Conda environment, activating it using the conda activate command in the terminal.

pip install git+https://github.com/SeBassTian23/Visual-Phenomics-Python.git --upgrade --no-cache-dir

Install package from local source by downloading the package and installing it manually using the command below.

pip install .

---

Getting started

Once the package is installed, you can import text files generated by Visual Phenomics into a DataFrame. Make sure that, in case you import multiple experiments, the times and light intensities match up between experiments. By default, the text files have an all prefix in the filenames. Since the filename is also the parameter name (or column name), the all gets removed by default. In case the prefix is different, use the prefix parameter to define a custom one to remove.

Import Dependency

import visual_phenomics_py as vppy

Check Package Version

vppy.version()

Import Experiments

You can import Visual Phenomics data from a single or multiple folders. To avoid re-importing the same files and even redo calculations, use the save and load functions to save the DataFrame at any point (see below).

Import Examples:

## Import files from a single folder
df = vppy.dataframe('./path/to/experiment-data')

## Import files from a single folder and remove prefix (e.g. MyData_phi2.txt)
df = vppy.dataframe('./path/to/experiment-data', prefix="MyData_")

## Import files from multiple folders
df = vppy.dataframe(['./path/to/experiment_01','./path/to/experiment_02'])

## Import files from multiple folders and remove different prefixes
## The string handed to prefix is parsed as a regular expression.
df = vppy.dataframe(['./path/to/experiment_01','./path/to/experiment_02'], prefix="MyData01_|MyData02_")

Note: When importing multiple folders, an additional categorical column will be added to the dataframe named folder which contains the import paths and allow to distinguish the data from individual folders.

Additional Functions

Dataframe Info

Information about the DataFrame. This includes columns, data types and memory consumption.

## DataFrame info using vppy
vppy.info(df)

## DataFrame info using pandas
df.info(memory_usage='deep')

Sample Names

This function returns a list of unique sample names found within the experiment or experiments.

samples = vppy.samples(df)

DataFrame Description

This function returns a worded description of the DataFrame content in regards to the experiment(s).

vppy.description(df)

Formatted Column Labels

The DataFrame column names can be returned as a formatted string to be used in matplotlib plots.

## Return a formatted string
vppy.label('phi2') 

### returns the string '$\Phi_{II}$'

## Return a formatted string
vppy.label('phi2', {'phi2': 'Y(II)' }) 

### returns the string 'Y(II)'

Plot Data

This function allow to quickly plot a single parameter versus time. If needed, the values for each sample can be averaged and the standard deviation is indicated as well. If needed, also only specific days can be selected to be plotted.

plot(df=None, param=None, *, avg=False, err='sem' days=[])

Examples for plotting:

## Plot individual samples for the parameter Phi2
vppy.plot(df, 'phi2')

## Plot averaged values for samples for the parameter Phi2
vppy.plot(df, 'phi2', avg=True)

## Plot averaged values for samples for the parameter Phi2 with standard deviation
vppy.plot(df, 'phi2', avg=True, err='std')

## Plot averaged values for samples for the parameter Phi2 only for two specific days
vppy.plot(df, 'phi2', avg=True, days=[2,3])

## Plot averaged values for samples Col-0 and Strain A
vppy.plot(df[df['name'].isin(['Col-0','Strain A'])], 'phi2', avg=True)

## Plot heat map for the parameter Phi2
vppy.heatmap(df, 'phi2')

## Plot heat map for parameter Phi2 for samples Col-0 and Strain
vppy.heatmap(df[df['name'].isin(['Col-0','Strain A'])], 'phi2')

## Plot heat map for the parameter Phi2 only for two specific days
vppy.heatmap(df, 'phi2', days=[2,3])

The light intensities defined and used in the experiment can be plotted in a single plot.

## Plot individual samples for the parameter Phi2
vppy.plot_light(df)

Calculations

Based of the available data imported into the dataframe, parameters can be calculated or re-calculated. Two functions are available. One to calculate parameters from the basic parameters directly derived from the images using Visual Phenomics and parameters, that are based on additional information like light intensity. The other one is for calculations based on parameters returned by the fist one.

Basic Calculations

For the calculation of basic parameters, the following parameters are available: Fvfm, NPQ, NPQt, Phi2, PhiNO, PhiNOt, PhiNPQ, PhiNPQt, qE, qEsv, qEt, qI, qIt, qL, and qP.

calculate(df=None, param='', *, fm='fm', f0='f0', fmp='fmp', f0p='f0p', fs='fs', fmpp='fmpp', f0pp='f0pp', fmf0=4.88, alias=None)

Examples for calculations:

# Calculating Phi2
vppy.calculate(df,'Phi2')

# Calculating Phi2 and redefine the used column names
vppy.calculate(df,'Phi2', fmp='FMP')

# Calculating Phi2 and redefine the value for Fm/F0 (default 4.88)
vppy.calculate(df,'Phi2', fmf0=4.0)

# Calculating Phi2 and renaming the column returned
vppy.calculate(df,'Phi2', alias='YII')

Additional Calculations

These additional calculations are for parameters that were calculated using the parameters returned by the basic calculation function. The parameters include LEF, Vx, SPhi2, SNPQ, and deltaNPQ.

calculate_additional(df=None, param='', *, v_phino='PhiNOt', v_phi2='Phi2', v_ql='qL', v_par='light_intensity', phinoopt=0.2, absorptivity=0.5, fmf0=4.88, alias=None)

Examples for calculations:

# Calculating LEF
vppy.calculate_additional(df,'LEF')

# Calculating LEF and redefine the used column names
vppy.calculate_additional(df,'LEF', v_phi2='YII')

# Calculating LEF and redefine the value for absorptivity (default 0.5)
vppy.calculate_additional(df,'LEF', absorptivity=0.45)

# Calculating LEF and renaming the column returned
vppy.calculate_additional(df,'LEF', alias='PPFD')

Custom Calculations

It also allows to create custom functions and apply the calculations to a dataframe column.

calculate_custom(df=None, name='', fn=None , *, cols=[], fill=[], params={})

Examples for calculations:

## Function with not parameters
def func():
  return 'Hello World'

vppy.calculate_custom(df, 'CustomFn', func )

## Function requiring data from columns to calculate Phi2
def func( fmp, fs ):
  return (fmp - fs) / fmp

vppy.calculate_custom(df, 'CustomPhi2', func, cols=['fmp', 'fs'] )

## Function requiring data from columns and parameters to calculate LEF
def func( fmp, fs, light, absorptivity=0.5 ):
  return ( (fmp - fs) / fmp ) * light * absorptivity

vppy.calculate_custom(df, 'CustomLEF', func, cols=['fmp', 'fs', 'light_intensity'], params={'absorptivity': 0.45} )

## Function requiring values from columns that doesn't contain a value for every timepoint.
## In this case FM and F0 are only available for the first timepoint and the nan values are
## filled using the pandas fillan(method-"ffill") function. The fill is temporary and only
## available during the calculation.
def func( npq, ql, fm, f0 ):
  return 1 / (npq + (1 + (ql * ((fm/f0)-1))))

vppy.calculate_custom(df, 'CustomPhiNO', func, cols=['npq', 'ql', 'fm', 'f0'], fill=['fm','f0'])

Utilities

Using the util sub-module, the functions used to calculate values for the whole dataframe, can be used to make calculations for an individual value.

## Calculating Phi2 for a single set of Fm' and and Fs values 
phi2 = vppy.util.phi2( 1300 , 669) #returns 0.486

## Now calculating the LEF parameter using the previously calculated Phi2 and the light intensity
lef = vppy.util.lef(phi2, 500) # returns 121.5

Currently the functions, fvfm, npq, npqt, phi2, phino, phinot, phinpq, phinpqt, qe, qesv, qet, qi, qit, ql, qp, and lef are available in the util sub-module.

Protocol Timing

The timing of DEPI protocols can fluctuate by seconds or fractions of seconds, which is reflected in the timing information in the Visual Phenomics output. In order to correct the fluctuations and allow easier comparison between experiments the original protocol timing can be generated.

## Generate Timing for standard DEPI sinusoidal 16h protocol
vppy.util.protocol_std_timing(protocol='sinusoidal')

## Generate Timing for standard DEPI sinusoidal 12h protocol
vppy.util.protocol_std_timing(hours=12, protocol='sinusoidal')

## Generate Timing for standard DEPI sinusoidal 12h protocol offset by 24h (2nd day)
vppy.util.protocol_std_timing(offset=24, hours=12, protocol='sinusoidal')

## Generate Timing for standard 16h day DEPI protocol consisting of a 
## flat, sinusoidal, fluctuating, flat and fluctuating day.
arr = np.concatenate(
  [
    vppy.util.protocol_std_timing(offset=0, protocol='flat'),
    vppy.util.protocol_std_timing(offset=24, protocol='sinusoidal'),
    vppy.util.protocol_std_timing(offset=48, protocol='fluctuating'),
    vppy.util.protocol_std_timing(offset=72, protocol='flat'),
    vppy.util.protocol_std_timing(offset=96, protocol='fluctuating')
  ],
  axis=0
)

## Generate Timing for standard 16h day DEPI protocol for parameters collected
## in the dark at the beginning of a day.
arr_dark = np.concatenate(
  [
    vppy.util.protocol_std_timing(offset=0, protocol='dark'),
    vppy.util.protocol_std_timing(offset=24, protocol='dark'),
    vppy.util.protocol_std_timing(offset=48, protocol='dark'),
    vppy.util.protocol_std_timing(offset=72, protocol='dark'),
    vppy.util.protocol_std_timing(offset=96, protocol='dark')
  ],
  axis=0
)

## Generate header line for output files by Visual Phenomics
vppy.util.vp_file_header(arr, initCol=True) # returns: name[position][flat][experiment][camera][replicate] 0.000  1.000  2.000...

## Generate header line for output files by Visual Phenomics
vppy.util.vp_file_header(arr_dark, initCol=False) # returns: 0.000  24.000  48.000  72.000  96.000

Backup and Export

The data represented inside the DataFrame can be exported back into individual text files, having the same format as the files provided by Visual Phenomics. The to_txt function for exporting is rather slow. The DataFrame itself can be saved using the save and also restored using the load functions. This is much faster than exporting the content to text files.

Export Parameters as Text Files

Each column within the DataFrame can be exported (with some exceptions) into a textfile, that has the same format as the data files provided by Visual Phenomics.

to_txt(df=None, folder=None, cols=[])

Examples:

## Export all parameters
vppy.to_txt(df, './export/')

## Export specific parameters
vppy.to_txt(df, './export/', cols=['phi2','fmp','fs'])

Save and Load a DataFrame

The DataFrame can be saved to a file at any time and this file can also be loaded as a DataFrame.

## To Save:
save(df=None, path=None, compression='zip')

## To Load:
load(filepath=None, compression='zip')

## Save DataFrame
vppy.save(df, './export')

## Load DataFrame
df = vppy.load('./export/dataframe.pkl')