python3

When /usr/bin/env: python: No such file or directory

/usr/bin/env: python: No such file or directory

sudo apt install python-is-python3
# or 
sudo ln -s /usr/bin/python3 /usr/bin/python

parsing example to match simple pattern

from parse import parse as p
dev_name = p("/dev/{}", ("/dev/loop0").strip(), case_sensitive=True)

May be you will get : ModuleNotFoundError: No module named 'parse'

zip'ed enumerated index on a list

L = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h']
for (idx, i) in zip(range(0, len(L)), L): 
    print(i, e)

Generate a list of all zeros

https://stackoverflow.com/questions/6076270/lambda-function-in-list-comprehensions https://stackoverflow.com/questions/10712002/create-an-empty-list-with-certain-size-in-python

This inits a list with the sequence of (0, 10) and then applies the lambda function which makes it zero (or any constant that we want)

# Make a null list 
xs = [None] * 10
# set all to zero 
[(lambda x: 0)(x) for x in [None] * 10]
# or an int list then square them 
last_values = [(lambda x: x*x)(x) for x in range(10)]

Formatting strings

https://docs.python.org/3/library/string.html#formatspec

'{:<30}'.format('left aligned')
'left aligned                  '
'{:>30}'.format('right aligned')
'                 right aligned'
'{:^30}'.format('centered')
'           centered           '
'{:*^30}'.format('centered')  # use '*' as a fill char
'***********centered***********'

I think {1} just meant the second argument in the parsing! Positional argument access.

'{0}, {1}, {2}'.format('a', 'b', 'c')
'a, b, c'
'{}, {}, {}'.format('a', 'b', 'c')  # 3.1+ only
'a, b, c'
'{2}, {1}, {0}'.format('a', 'b', 'c')
'c, b, a'
'{2}, {1}, {0}'.format(*'abc')      # unpacking argument sequence
'c, b, a'
'{0}{1}{0}'.format('abra', 'cad')   # arguments' indices can be repeated
'abracadabra'

What does __name__ == "__main__" do?

Special direct invocation guard

What does if name == "main": do?

Good explanation of subprocess interface of python

https://stackoverflow.com/questions/4256107/running-bash-commands-in-python

cmd = '''while read -r x;
   do ping -c 3 "$x" | grep 'min/avg/max'
   done <hosts.txt'''

# Trivial but horrible
results = subprocess.run(
    cmd, shell=True, universal_newlines=True, check=True)
print(results.stdout)

# Reimplement with shell=False
with open('hosts.txt') as hosts:
    for host in hosts:
        host = host.rstrip('\n')  # drop newline
        ping = subprocess.run(
             ['ping', '-c', '3', host],
             text=True,
             stdout=subprocess.PIPE,
             check=True)
        for line in ping.stdout.split('\n'):
             if 'min/avg/max' in line:
                 print('{}: {}'.format(host, line))

What is drgn

Used in the /tools/workqueue scripts from https://www.kernel.org/doc/html/next/core-api/workqueue.html#examining-configuration and https://github.com/iovisor/bcc/blob/master/tools/wqlat_example.txt

sudo apt-get install python3-drgn 

https://drgn.readthedocs.io/en/latest/ https://github.com/osandov/drgn

drgn was developed at Meta for debugging the Linux kernel (as an alternative to the crash utility), but it can also debug userspace programs written in C. C++ support is in progress.

dict vs. list

See the difference between {} and []

>>> result:dict[str, int] = {}
>>> type(result) 
<class 'dict'>
>>> result:dict[str, int] = []
>>> type(result) 
<class 'list'>
>>> 

C-like structure

https://stackoverflow.com/questions/35988/c-like-structures-in-python

@dataclass
class Point:
    x: float
    y: float
    z: float = 0.0

p = Point(1.5, 2.5)
print(p)  # Point(x=1.5, y=2.5, z=0.0)

immutable:

from typing import NamedTuple

class User(NamedTuple):
    name: str

class MyStruct(NamedTuple):
    foo: str
    bar: int
    baz: list
    qux: User

my_item = MyStruct('foo', 0, ['baz'], User('peter'))
print(my_item) # MyStruct(foo='foo', bar=0, baz=['baz'], qux=User(name='peter'))

Zip returns an iterator, not a list

https://stackoverflow.com/questions/17777219/zip-variable-empty-after-first-use

hence after the first use the iterator is empty.

How to generate timestamps

https://strftime.org/

>>> from datetime import datetime
>>> datetime.today().strftime('%Y-%m-%d')
'2021-01-26'

>>> datetime.today().strftime('%Y-%m-%d %H:%M:%S')
'2021-01-26 16:50:03'

Python string formatting

• https://realpython.com/python-f-strings/
• https://realpython.com/python-print/
• https://docs.python.org/3/tutorial/introduction.html

July 19th

Following the instructions from https://github.com/stonet-research/icpe24_io_scheduler_study_artifact/blob/main/fig-1-samsung-baseline/qd_iops_vary_bs.py then ran into the issues externally managed issue error: externally-managed-environment

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your 
Python installation or OS distribution provider. You can override 
this, at the risk of breaking your Python installation or OS, 
by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Solution is to setup the venv and use that, see this. Sequence of commands:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt

sudo access issue https://discuss.python.org/t/using-sudo-inside-venv/29148/2 (take the full path)

(.venv) $ which python3 
/home/atr/src/.venv/bin/python3
(.venv) $ sudo which python3 
/usr/bin/python3
(.venv) $ sudo .venv/bin/python3 test.py -r --fio ~/src/fio/fio --dev /dev/nvme0n1

python-fio workflow

Regarding how to use Python for fio and plotting, I have a couple of general pointers.

Packaging/dependencies

I make use of venv, https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/. This ensures pip3 only installs packages locally. For each Python project I first setup venv within the top-level of the repository:

python3 -m venv .venv
echo .venv >> .gitignore # Should be in the ignore to prevent pushing a vm to git

Whenever, you then want to work on the project, you first type (can be automated with direnv):

source ./venv/bin/activate

You can move your locally installed packages to a dependency file with:

pip3 freeze > requirements.txt

And install them with:

pip3 install -r requirements.txt

All other "suggested" alternatives like poetry and pipenv are slow, made for production software and break every single month.

Mandatory packages required for a good experience

pip3 install matplotlib numpy notebook

Matplotlib is the standard for plotting (https://matplotlib.org/, a good addition is Seaborne) Numpy installs the standard numerics/data structures library used in data science (https://numpy.org/, a friendlier interface on top is https://pandas.pydata.org/). Notebook installs Jupyter (https://docs.jupyter.org/en/latest/)

How to read a json file

import json

with open('fio_data.json') as f:
    d = json.load(f)
    print(d)

How to use matplotlib

The default guide is alright: https://matplotlib.org/stable/users/explain/quick_start.html

How to use notebooks

If notebooks or jupyter lab is installed, you can type in the terminal:

jupyter notebook

It should automatically open your browser.