Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More index optimisations #923

Merged
merged 20 commits into from
Feb 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
250b518
Indices: Initial work to support calc_abc shortcut
shawnlaffan Feb 24, 2024
c2741dc
Indices metadata: better feedback on error
shawnlaffan Feb 24, 2024
17d0834
Indices: fix cache logic in _calc_abc_any, start using it
shawnlaffan Feb 24, 2024
8b2e965
Indices precalcs: more now use _calc_abc_any
shawnlaffan Feb 24, 2024
1acc8f5
Indices: calc_matrix_stats now uses _calc_abc_any
shawnlaffan Feb 24, 2024
a9c60e7
Indices: several phylo calcs now depend on _calc_abc_any
shawnlaffan Feb 24, 2024
edde81d
Indices: calc_labels_not_on_trimmed_tree now uses _calc_abc_any
shawnlaffan Feb 24, 2024
3fe31fd
Indices::Rarity: Simplify inheritance and some metadata
shawnlaffan Feb 24, 2024
fd012ef
Indices: revert calc_labels_on_tree dep back to calc_abc
shawnlaffan Feb 25, 2024
17b477e
Indices: calc_abc adapts cached calc_abc2 or calc_abc3 results
shawnlaffan Feb 25, 2024
466eeaa
Indices: ensure calc_abc2 and 3 are run before calc_abc
shawnlaffan Feb 25, 2024
b7456f0
Indices: make AS_RESULTS_FROM_LOCAL a proper param, not a cache entry
shawnlaffan Feb 25, 2024
b321c48
Indices: optimise _calc_phylo_abc_lists
shawnlaffan Feb 25, 2024
496687a
Indices: minor refactor of _calc_rarity_(central|whole)
shawnlaffan Feb 25, 2024
0aa0afd
Indices: rarity whole and central copy results when needed
shawnlaffan Feb 25, 2024
2cd3978
Indices: calc_phylo_rpe_central can copy calc_phylo_rpe2 if no set2 nbrs
shawnlaffan Feb 25, 2024
71ef863
Indices: rpe_central and rpe2 can share results if no nbr set2
shawnlaffan Feb 25, 2024
c1a3f9d
Indices: micro-optimise the aed calcs
shawnlaffan Feb 25, 2024
f1d872d
Indices: add a hierarchical calc_abc variant
shawnlaffan Feb 25, 2024
a046fe1
rand reintegration tests: change numeric tolerance to 1e-8
shawnlaffan Feb 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 22 additions & 3 deletions lib/Biodiverse/Indices.pm
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ use warnings;
#use Devel::Symdump;
#use Data::Dumper;
use Scalar::Util qw /blessed weaken/;
use List::MoreUtils qw /uniq/;
use List::MoreUtils qw /uniq first_index/;
use List::Util qw /sum any/;
use English ( -no_match_vars );
use Ref::Util qw { :all };
Expand Down Expand Up @@ -1021,6 +1021,25 @@ sub aggregate_calc_lists_by_type {
foreach my $type (@types) {
my $array = $aggregated{$type};
my @u_array = uniq @$array;
if ($type eq 'pre_calc'
and scalar @u_array
) {
# move first /calc_abc[23]/ to front so
# calc_abc and _calc_abc_any can grab results
# otherwise ensure calc_abc is at the front
# for _calc_abc_any
state $re = qr{^calc_abc[23]};
my $iter23 = first_index {$_ =~ $re} @u_array;
if ($iter23 > 0) {
unshift @u_array, splice @u_array, $iter23, 1;
}
else {
my $iter1 = first_index {$_ eq 'calc_abc'} @u_array;
if ($iter1 > 0) {
unshift @u_array, splice @u_array, $iter1, 1;
}
}
}
$aggregated{$type} = \@u_array;
}

Expand Down Expand Up @@ -1533,7 +1552,7 @@ sub run_dependencies {
my %results;
my %as_results_from;
# make sure this is new each iteration
$self->set_cached_value ($cache_name_local_results => \%as_results_from);
$self->set_param ($cache_name_local_results => \%as_results_from);

foreach my $calc (@$calc_list) {
my $calc_results;
Expand Down Expand Up @@ -1572,7 +1591,7 @@ sub run_dependencies {
}

# We refresh each call above, but this ensures last one is cleaned up.
$self->delete_cached_value($cache_name_local_results);
$self->delete_param ($cache_name_local_results);

if ( $type eq 'pre_calc_global' ) {
$self->set_param( AS_RESULTS_FROM_GLOBAL => \%as_results_from_global );
Expand Down
11 changes: 6 additions & 5 deletions lib/Biodiverse/Indices/Endemism.pm
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ sub calc_endemism_central_hier_part {
# If we have no nbrs in set 2 then we are the same as the "whole" variant.
# So just grab its values if it has already been calculated.
if (!keys %{$args{label_hash2}}) {
my $cache_hash = $self->get_cached_value('AS_RESULTS_FROM_LOCAL');
my $cache_hash = $self->get_param('AS_RESULTS_FROM_LOCAL');
my $cached = $cache_hash->{calc_endemism_whole_hier_part};
if ($cached) {
my %remapped;
Expand Down Expand Up @@ -335,7 +335,7 @@ sub calc_endemism_whole_hier_part {
# If we have no nbrs in set 2 then we are the same as the "central" variant.
# So just grab its values if it has already been calculated.
if (!keys %{$args{label_hash2}}) {
my $cache_hash = $self->get_cached_value('AS_RESULTS_FROM_LOCAL');
my $cache_hash = $self->get_param('AS_RESULTS_FROM_LOCAL');
my $cached = $cache_hash->{calc_endemism_central_hier_part};
if ($cached) {
# say STDERR join ' ', sort keys %$cached;
Expand Down Expand Up @@ -586,7 +586,7 @@ sub _calc_endemism_central {
# If we have no nbrs in set 2 then we are the same as the "whole" variant.
# So just grab its values if it has already been calculated.
if (!keys %{$args{label_hash2}}) {
my $cache_hash = $self->get_cached_value('AS_RESULTS_FROM_LOCAL');
my $cache_hash = $self->get_param('AS_RESULTS_FROM_LOCAL');
my $cached = $cache_hash->{_calc_endemism_whole};
return wantarray ? %$cached : $cached
if $cached;
Expand Down Expand Up @@ -736,7 +736,7 @@ sub _calc_endemism_whole {
# If we have no nbrs in set 2 then we are the same as the "central" variant.
# So just grab its values if it has already been calculated.
if (!keys %{$args{label_hash2}}) {
my $cache_hash = $self->get_cached_value('AS_RESULTS_FROM_LOCAL');
my $cache_hash = $self->get_param('AS_RESULTS_FROM_LOCAL');
my $cached = $cache_hash->{_calc_endemism_central};
return wantarray ? %$cached : $cached
if $cached;
Expand All @@ -761,7 +761,8 @@ sub _calc_endemism {
? $args{label_hash1}
: $args{label_hash_all};

# allows us to use this for any other basedata get_* function
# Allows us to use this for any other basedata get_* function.
# calc_rarity is an example of this.
my $function = $args{function} || 'get_range';
my $range_hash = $args{label_range_hash} || {};

Expand Down
Loading
Loading