Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modernize portfolio optimizer #376

Merged
merged 18 commits into from
Jun 20, 2024
Merged

Modernize portfolio optimizer #376

merged 18 commits into from
Jun 20, 2024

Conversation

jtao1
Copy link
Collaborator

@jtao1 jtao1 commented Apr 3, 2024

Modernizing an example checklist

Preliminary checks

  • Look for open PRs and issues that reference the project you are updating. It is possible previous unmerged work in PR could be re-used to modernize the project. Comment on these PRs and issues when appropriate, hopefully we should be able to close some of them after your modernizing work.

Change ‘anaconda-project.yml’ to use the latest workable version of packages

  • Pin python=3.11
  • Remove the upper pin (e.g. hvplot<0.9 to hvplot, panel>=0.12,<1.0 to panel>=0.12) of all other dependencies. Removing the upper pins of dependencies could necessitate code revisions in the notebooks to address any errors encountered in the updated environment. Should complexities or extensive time requirements arise, document issues for team discussion on whether to re-pin specific packages or explore other solutions.
  • Add/update the lower pin of all other dependencies (e.g. hvplot to hvplot>=0.9.2, hvplot>=0.8 to hvplot>=0.9.2). Usually, the new/updated lower pin of a dependency will be the version resolved after anaconda prepare has been run. Execute !conda list in a notebook, or anaconda run conda list in the terminal, to display the version of each dependency installed in the environment. Adjusting the lower pin helps ensure that the locks produced for each platform (linux-64, win-64, osx-64, osx-arm64) rely on the tested dependencies and not on some older versions.
  • If one of the channels include conda-forge or pyviz, ask Maxime if it can be removed

Plot API updates (discussed on a per-example basis)

  • Generally, try to replace HoloViews usage with hvPlot. At a certain point of complexity, such as with the use of ‘.select’, it might be better to stick with HoloViews. Additional examples of ‘complexity boundaries’ should be documented in this document.
  • Almost always, try to replace the use of datashade with rasterize (read this page). Essentially, rasterize allows Bokeh to handle the colormapping instead of Datashader.

Interactivity API updates (discussed on a per-example basis)

  • Remove all pn.interact usage
  • Avoid .param.watch() usage. This is pretty low-level and verbose approach and should not be used in Examples unless required, or an Example is specifically trying to demo its usage in an advanced workflow.
  • Prefer using pn.bind(). Read this page for explanation.
  • For apps built using a class approach, when they create a view() method and call it directly, update the class by inheriting from pn.viewable.Viewer and replace view() by __panel__(). Here is an example.

Panel App updates (discussed on a per-example basis)

  • If the project doesn’t at any point create a Panel app at all, consider creating one. It can be as simple as wrapping a plot in pn.Column, or more complicated to incorporate widgets, etc. Make the final app .servable().
  • If the project creates an app in a notebook but doesn’t deploy it (i.e. there is no command: dashboard declaration in the anaconda-project.yml file), try adding it.
  • If the project already deploys an app but doesn’t wrap it in a nice template, consider wrapping it in a template.
  • If the project deploys an app wrapped in a template, customize the template a little so all the apps don’t look similar (e.g. change the header background color). This doesn’t need to be discussed.
  • Comment start If you are building the application in a single cell, you can construct a template explicitly, like template = pn.template.BootstrampTemplate, but if building up an app across multiple cells, it is probably cleaner to declare the template at the top with pn.extension(template='bootstrap'). See how to guide on setting a template.

General code quality updates

  • If the notebook disables warnings (e.g. with warnings.simplefilter(‘ignore’) somewhere at the start of the notebook, remove this line. Try to update the code to remove the warnings, if any. If updating the code to remove the warnings is taking significant amount of time and effort, bring it up for discussion and we may decide to disable warnings again.

Text content

  • Edit the text content anywhere and everywhere that it can be improved for clarity.
  • Check the links are valid, and update old links (e.g. http -> https, xyz.pyviz.org -> xyz.holoviz.org)
  • Remove instructions to install packages inside an example

Visual appearance - Example

  • Check that the titles/headings make sense and are succinct.
  • Check that the text content blocks are easily readable; revise into additional paragraphs if needed.
  • Check that the code blocks are easily readable; revise as needed. (e.g. add spaces after commas in a list if there are none, wrap long lines, etc.)
  • Check image and plot sizes. If possible, making them responsive is highly recommended.
  • Check the appearance on a smartphone (check Google to see how to adapt the appearance of your browser to display pages as if they were seen from a smartphone, this is usually done via the web developer tools). This is not a top priority for all examples, but if there are a few easy and straightforward changes to make that can improve the experience, let’s do it.
  • Check the updated notebook with the original notebook

Visual appearance - Gallery

  • Check the thumbnail is visually appealing
  • Check the project title is well formatted (e.g. Ml Annotators to ML Annotators), if not, add/update the examples_config.title field in anaconda-project.yml
  • Check the project description is appropriate, if not, update the description field in anaconda-project.yml

Workflow (after you have made the changes above)

  • Run successfully doit validate:<projectname>
  • Run successfully doit test:<projectname>
  • Run successfully doit doc_one –name <projectname>. It’s better if the project notebook(s) is saved with its outputs (but be sure to clear outputs before committing to the examples repo!) when building the docs. Then open this file in your browser ./builtdocs/index.html and check how the site looks.
  • If you’re happy with all the above, open a PR. Reminder, clear notebook outputs before pushing to the PR.

@jtao1 jtao1 self-assigned this Apr 3, 2024
Copy link
Contributor

github-actions bot commented Apr 3, 2024

Your changes were successfully integrated in the dev site, make sure to review
the pages of the projects you touched before merging this PR: https://holoviz-dev.github.io/examples/.
You can also download an archive of the site from the workflow summary page which comes in handy
when your dev site built was overriden by another PR (we have a single dev site!).

@Azaya89 Azaya89 added the NF SDG NumFocus Software Development Grant 2024 label Apr 3, 2024
@jtao1 jtao1 requested a review from maximlt April 10, 2024 05:28
Copy link
Contributor

@maximlt maximlt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed the PR. Before adding general comments, I wanted to ask you the status of these tasks you wanted to complete?

Submitted first update on portfolio optimizer. Some things that I still need to do:

* added better explanations for the code especially the newer ones related to creating the panel app

* remove possibly redundant math when creating the graphs

Comments after review:

  • It took me a little while to get what 252 stood for, it turns out it's the number of trading days in a year. Make this a little clearer in the notebook please.
  • The first part of the example, that was already there, is quite dry. I don't think spending time improving that is really in the scope of our work, unless that's something that would be quick and easy for you to do!
  • However, I think the introduction could be improved. At the moment it only talks about the financial aspect of the example, it could be extended a little to describe the HoloViz tools that are going to be used (hvplot, holoviews, panel) and how they support the analysis. It should also be mentioned a Panel dashboard is going to be created. Adding a screenshot of this app at the beginning would be nice (check out Isaiah's PR on nyc_taxi and the start of the dashboard.ipynb notebook).
  • My main feedback: the section you have ported from Panel's website is also very dry. We need to add some explanation about the app we are building and how we are building it. The Build a <app> tutorials you have recently gone through could be a good template for this.
  • We should add this app as a deployment (check out ./template/anaconda-project.yaml to see how). And if we do that, we could also wrap it in a template at the very end of the notebook, so the deployed app looks good.
  • Some of the elements of the app could be made responsive (i.e. set up so they dynamically stretch to occupy the available space).
  • The app is built using hvPlot's interactive API. It's possible we'll try to migrate to pn.rx. I'm waiting for feedback from others on Discord on this point.

EDIT: I updated your original post with the modernizing checklist.

portfolio_optimizer/portfolio_optimizer.ipynb Outdated Show resolved Hide resolved
portfolio_optimizer/portfolio_optimizer.ipynb Outdated Show resolved Hide resolved
@maximlt
Copy link
Contributor

maximlt commented Apr 18, 2024

I have a suggestion about the notebook being a little dry and lacking context. Maybe you could throw the notebook content at ChatGPT and ask for it to add some brief explanations about the underlying analysis. That's just a suggestion, so feel free to ignore it if you don't feel like it. I think though that ChatGPT could do a good job at this, given this is I believe a very common kind of analysis.

Copy link
Contributor

Your changes were successfully integrated in the dev site, make sure to review
the pages of the projects you touched before merging this PR: https://holoviz-dev.github.io/examples/.
You can also download an archive of the site from the workflow summary page which comes in handy
when your dev site built was overriden by another PR (we have a single dev site!).

@jbednar
Copy link
Contributor

jbednar commented Apr 29, 2024

It's possible we'll try to migrate to pn.rx

Is there anyone who is arguing for keeping it hvplot.interactive? pn.rx seems like the obvious thing to do, unless someone is very convincingly arguing otherwise.

@ahuang11
Copy link
Contributor

ahuang11 commented May 14, 2024

Was helping Jason with rx and it felt like rx was a bit magical / unintuitive to me.

Also, we weren't sure how to separate out the widgets to better lay out the dashboard, unlike pn.Param.

I'm -1 on using rx since I think it could just potentially confuse new users with all the ways to do something--at least until there are more examples and docs on how to use it, i.e. do this/not that, good patterns/anti-patterns.

from io import BytesIO
import pandas as pd
import panel as pn
pn.extension()

@pn.cache
def get_stocks(data):
    if data is None:
        stock_file = 'https://datasets.holoviz.org/stocks/v1/stocks.csv'
    else:
        stock_file = BytesIO(data)
    return pd.read_csv(stock_file, index_col='Date', parse_dates=True)


file_input = pn.widgets.FileInput(sizing_mode='stretch_width')

stocks = pn.rx(get_stocks)(file_input)

selector = pn.widgets.MultiSelect(
    name='Select stocks', sizing_mode='stretch_width',
    options=stocks.columns.to_list()
)

selected_stocks = stocks.rx.pipe(
    lambda df, cols: df[cols] if cols else df, selector
)

pn.Column(selected_stocks)
image

Lastly, upon printing rx objects, it raises an error
holoviz/param#939

@ahuang11
Copy link
Contributor

ahuang11 commented May 14, 2024

Ok I just realized there's a section in Panel about rx; I was too focused on the param docs.

To split the widget + column, you need to wrap it in the corresponding pane; personally I'd prefer a method to call to do so, e.g. selected_stocks.get_pane()

from io import BytesIO
import pandas as pd
import panel as pn
pn.extension()

@pn.cache
def get_stocks(data):
    if data is None:
        stock_file = 'https://datasets.holoviz.org/stocks/v1/stocks.csv'
    else:
        stock_file = BytesIO(data)
    return pd.read_csv(stock_file, index_col='Date', parse_dates=True)


file_input = pn.widgets.FileInput(sizing_mode='stretch_width')

stocks = pn.rx(get_stocks)(file_input)

selector = pn.widgets.MultiSelect(
    name='Select stocks', sizing_mode='stretch_width',
    options=stocks.columns.to_list()
)

selected_stocks = stocks.rx.pipe(
    lambda df, cols: df[cols] if cols else df, selector
)

pn.Column(
    pn.Row(file_input, selector),
    pn.pane.DataFrame(selected_stocks),
)

Copy link
Contributor

Your changes were successfully integrated in the dev site, make sure to review
the pages of the projects you touched before merging this PR: https://holoviz-dev.github.io/examples/.
You can also download an archive of the site from the workflow summary page which comes in handy
when your dev site built was overriden by another PR (we have a single dev site!).

Copy link
Contributor

Your changes were successfully integrated in the dev site, make sure to review
the pages of the projects you touched before merging this PR: https://holoviz-dev.github.io/examples/.
You can also download an archive of the site from the workflow summary page which comes in handy
when your dev site built was overriden by another PR (we have a single dev site!).

portfolio_optimizer/portfolio_optimizer.ipynb Outdated Show resolved Hide resolved
portfolio_optimizer/portfolio_optimizer.ipynb Show resolved Hide resolved
portfolio_optimizer/portfolio_optimizer.ipynb Outdated Show resolved Hide resolved
portfolio_optimizer/portfolio_optimizer.ipynb Outdated Show resolved Hide resolved
portfolio_optimizer/portfolio_optimizer.ipynb Outdated Show resolved Hide resolved
@jbednar
Copy link
Contributor

jbednar commented May 22, 2024

@ahuang11 : Ok I just realized there's a section in Panel about rx; I was too focused on the param docs.

Can you please make a PR on Param putting in an explicit link to the Panel rx docs wherever you would have found them? Right now it only links to the main Panel site, but pointing directly to Panel rx info would presumably prevent others from having the same experience you did.

Copy link
Contributor

Your changes were successfully integrated in the dev site, make sure to review
the pages of the projects you touched before merging this PR: https://holoviz-dev.github.io/examples/.
You can also download an archive of the site from the workflow summary page which comes in handy
when your dev site built was overriden by another PR (we have a single dev site!).

Copy link
Contributor

github-actions bot commented Jun 5, 2024

Your changes were successfully integrated in the dev site, make sure to review
the pages of the projects you touched before merging this PR: https://holoviz-dev.github.io/examples/.
You can also download an archive of the site from the workflow summary page which comes in handy
when your dev site built was overriden by another PR (we have a single dev site!).

Copy link
Contributor

Your changes were successfully integrated in the dev site, make sure to review
the pages of the projects you touched before merging this PR: https://holoviz-dev.github.io/examples/.
You can also download an archive of the site from the workflow summary page which comes in handy
when your dev site built was overriden by another PR (we have a single dev site!).

@droumis
Copy link
Contributor

droumis commented Jun 20, 2024

I made a lot of changes. There were still many issues: headings, grammatical, organizational, textual. In the future, the PRs need to be further along prior to asking for a final review.

Copy link
Contributor

Your changes were successfully integrated in the dev site, make sure to review
the pages of the projects you touched before merging this PR: https://holoviz-dev.github.io/examples/.
You can also download an archive of the site from the workflow summary page which comes in handy
when your dev site built was overriden by another PR (we have a single dev site!).

@droumis droumis merged commit 08a5db9 into main Jun 20, 2024
9 checks passed
@maximlt maximlt deleted the modernize_portfolio_optimizer branch June 21, 2024 06:54
Azaya89 pushed a commit that referenced this pull request Jul 4, 2024
* added panel dashboard

* clear outputs

* added more text description to cells

* improved the introduction of the notebook

* removed param.watch

* cleared outputs

* added more cell descriptions

* fixed linter error

* updated holoview plots to hvplot

* removed outputs

* updated yml

* minor edits to made code more readable

* changes to cell descriptions

* updated yml files

* updated yaml files again

* dr revision of text code and viz

* color the distribution plots

---------

Co-authored-by: jtao1 <jtao@LAPTOP-UKS0PRO9>
Co-authored-by: Demetris Roumis <roumis.d@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NF SDG NumFocus Software Development Grant 2024
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants