DistributedArray

[mrava87]

This discussion is meant to discuss the design choices of the DistributedArray class.

All in all, I think it looks good and its design seems nice and clean. Also, you handle both of the scenarios we are interested in (global array divided into parts, and global array replicated to all ranks).

Here are few comments regarding the object itself (mostly stylistic):

• Why do you subclass np.ndarray? I assume pretty much all methods will not work as they are unless they are not overwritten with distributed logic? From what I understand, maybe this allows you not to reimplement operations like add, subtract as you work with only the numpy arrays in each rank?
• I suggest to default dtype to np.float64 to be consistent with what we do in pylops. I would also always put it as last in the order of inputs
• May be a good idea to add rank as member of the DistributedArray class, especially for debugging purposes. So this is not called directly (base_comm.Get_rank()) inside the to_dist method
• type_part is a bit of a strange name, I guess you could just call it partition as it took me a while to understand probably part was an abbreviation for partition
• Not sure I agree with meaning of scatter (S) in type_part. For me scatter means that one rank (say rank0) has an array and scatters it to all other ranks... here we are more creating an array that is naturally distributed and fill each portion in each rank directly. Maybe others have an idea for a better name.
• Currently, the below code returns an error. I think it is due to the fact that SumArray = DistributedArray(self.global_shape, dtype=self.dtype) does not set type_part. Maybe we should also include some logic that if the two arrays have different type_part an error is raised?

arr2 = DistributedArray(global_shape=(1000, 100), dtype=int, type_part="B")
arr3 = DistributedArray(global_shape=(1000, 100), dtype=int, type_part="B")
arr2[:] = 0
arr3[:] = 1
arr4 = arr2+arr2

A couple of more general comments:

• Would be good to already build your examples with comments providing a description of what you show (similar to pylops examples - eg https://github.com/PyLops/pylops/blob/dev/examples/plot_flip.py)
• It is important to build tests as you develop, so please always try to provide code with tests at the same time.

[rohanbabbar04]

• Agreed, made the DistributedArray not extend the np.ndarray and also made some changes to handle the same.
• Made the default dtype to np.float64, and is now at the last.
• Added rank and size as members for debugging.
• Changed the name from type_part to partition.
• Need a name suggestion for this one, currently it is still S and B.
• Made the required changes, now it works for both partitions(S and B), also corrected the sub.
• Added __neg__ , asarray(for global view of the array).
• Added some tests for the class DistributedArray. Use make tests to run the tests. Will be adding more...
• Will be adding a more descriptive example...

[mrava87]

@rohanbabbar04 a few comments from the latest version of DistributedArray:

• This code pattern is repeated a few times:

if self.partition != dist_array.partition:
            raise ValueError("Partition of both the arrays must be same")
if self.local_shape != dist_array.local_shape:
            raise ValueError(f"Local Array Shape Mismatch - "
                             f"{self.local_shape} != {dist_array.local_shape}")

maybe worth wrapping it into a method or auxiliary function

• Example: very nice. Only comment, I would avoid global_shape = (10, 10), having 10 for both dimensions may leave some bug in just because you could take either axis and would get 10... always better to use different numbers (same applies to tests, which I would also test with both even and odd numbers as sometimes there are edge cases that have to be treated differently...)

• dot method: I am not sure whether you are trying to mimic np.dot. Initially I just though this will be for 1d arrays but now since we have a container for Nd-arrays then I guess this is what you want to do? Also, it isn't obvious to me why you need those 3 checks to begin with. I would assume you need the same 2 as in the other methods but they do not seem to be there (can you handle arrays that have different partition?)

• norm method: are you implicitly assuming the norm is computed over the partitioned axis? If so, I am not sure I get this np.empty(shape=self.local_shape, dtype=self.dtype) as I would expect that if the array has global shape (N,M) and axis=0 the norm will have shape M but now it seems to be initialized to have it has shape (x, M) with x the size of the local array. Maybe I am not getting what you try to do :)