Enormous storage requirement.

[felme1]

Hello,
I am running the following code on a rocky linux machine. Everything seems to be working just fine, at least the data makes sense. However the memory required to run this simulation is already quite large. With is_3D set to False it used up more then 20GB of ram storage, and when I run it with is_3D set to True the used storage went up to 1.5 TB. Offcourse the runtime is very long as well. The 3d run took about 24 hours.

This seems crazy high to me, am I doing something wrong here?
The simulated volume is 33um by 10.5um by 6um with a resolution of 50. (This res is chosen as I would like to use dispersive materials) I understand that this might be quite a large simulation, but 1.5TB still seems quite excessive to me.

Ideally I would like to use the dispersive materials, but when I try to run the simulation using these materials the 2.2TB storage on this machine, that my university gave me access to is not enough.

Any help or clarification on this would be greatly appreciated!

Here is a picture of the simulation setup, so you can get a feeling for the gds file as well. I have also attatched the compressed gds file as well as the full code.

hybrid_coupler.zip
meep_sim_of_hybrid_coupler.zip

import meep as mp 
#from meep.materials import SiO2, Si
import numpy as np

class SimulationHandler:
    def __init__(self):
        self.Ex_list = []
        self.Ey_list = []
        self.Ez_list = []
        self.Hx_list = []
        self.Hy_list = []
        self.Hz_list = []

    def update_fields(self, ex, ey, ez, hx, hy, hz):
        self.Ex_list.append(ex[0])
        self.Ey_list.append(ey[0])
        self.Ez_list.append(ez[0])
        self.Hx_list.append(hx[0])
        self.Hy_list.append(hy[0])
        self.Hz_list.append(hz[0])
    
    def save_data(self, cell_x, cell_y):
        Ex = np.array(self.Ex_list)
        Ey= np.array(self.Ey_list)
        Ez = np.array(self.Ez_list)
        Hx = np.array(self.Hx_list)
        Hy= np.array(self.Hy_list)
        Hz = np.array(self.Hz_list)
        np.savez(f'updated/fieldData.npz', Ex=Ex, Ey=Ey, Ez=Ez, Hx=Hx, Hy=Hy, Hz=Hz, cell_x=cell_x, cell_y=cell_y)

res = 50  # pixels/μm
is_3D = True  # 3d calculation?
gdsII_file = "hybrid_coupler.gds" 

# Materials used in the simulation
Si = mp.Medium(index=3.45)
SiO2 = mp.Medium(index=1.444)

#dimensions of the coupler that is to be simulated
coupler_length = 28
coupler_width = 4.5
port_separation = 4
si_thickness = 0.22

#dimensions of the simulation
sim_thickness = 2
pml_thickness = 2

# Set up frequency points for simulation
npoints = 20
lcen = 1.55  #central wavelength
dwl = 0.05
fcen = 1 / lcen
df = 1 / dwl
df_source =  3 * dwl / lcen**2  # ONLY USE FOR SOURCES TO ENSURE LARGE ENOUGH FREQUENCY BANDWIDTH FOR SIMULATION
wl = np.linspace(lcen - dwl/2, lcen + dwl/2, npoints)
fpoints = 1 / wl


cell_thickness = pml_thickness + sim_thickness + pml_thickness if is_3D else 0
si_thickness = si_thickness if is_3D else 20

# Sets the min and max values for the cell and the silicon. Our simulation will be centered at y=0
si_zmax = 0.5 * si_thickness
si_zmin = -0.5 * si_thickness

#import geometry from gds file
geometry = mp.get_GDSII_prisms(Si, gdsII_file, 1, si_zmin, si_zmax)

#define simulation volume
cell = mp.Vector3(coupler_length + 2*pml_thickness + 1,coupler_width+2*pml_thickness + 2,cell_thickness)
cell_center = mp.Vector3(0,0,0)

#define location and geometry of source and monitors
port_size = mp.Vector3(0,1,1)
port_dx = coupler_length/2
port_dy = port_separation/2
port1 = mp.Volume(center=mp.Vector3(-port_dx,-port_dy,0), size=port_size)
port2 = mp.Volume(center=mp.Vector3(-port_dx,port_dy,0), size=port_size)
port3 = mp.Volume(center=mp.Vector3(port_dx,port_dy,0), size=port_size)
port4 = mp.Volume(center=mp.Vector3(port_dx,-port_dy,0), size=port_size)
source = mp.Volume(center=mp.Vector3(-port_dx-0.25,-port_dy,0),size=port_size)

mode_parity = mp.NO_PARITY if is_3D else mp.EVEN_Y + mp.ODD_Z

# setup source
sources = [
    mp.EigenModeSource(
        src=mp.GaussianSource(wavelength=lcen, fwidth=df_source),
        size=source.size,
        center=source.center,
        eig_band=1,
        eig_parity=mode_parity,
        eig_match_freq=True,
    )
]

#setup simulation
sim = mp.Simulation(
    resolution=res,
    cell_size=cell,
    boundary_layers=[mp.Absorber(pml_thickness)],
    sources=sources,
    geometry=geometry,
    default_material=SiO2
)


# Adds mode monitors at each of the ports to track the energy that goes in or out
mode_monitor_1 = sim.add_mode_monitor(fpoints, mp.ModeRegion(volume=port1))
mode_monitor_2 = sim.add_mode_monitor(fpoints, mp.ModeRegion(volume=port2))
mode_monitor_3 = sim.add_mode_monitor(fpoints, mp.ModeRegion(volume=port3))
mode_monitor_4 = sim.add_mode_monitor(fpoints, mp.ModeRegion(volume=port4))

plot_plane = mp.Volume(center=cell_center, size=mp.Vector3(cell.x, cell.y, 0))
sim.plot2D(output_plane=plot_plane if is_3D else None) # No parameters are needed for a 2D simulation. 
plt.savefig("updated/sim_setup.png")
#plt.show()


sim_handler = SimulationHandler()

def updateField(sim):
    ex = np.array([sim.get_array(center=cell_center,size=mp.Vector3(cell.x, cell.y, 0),component=mp.Ex)])
    ey = np.array([sim.get_array(center=cell_center,size=mp.Vector3(cell.x, cell.y, 0),component=mp.Ey)])
    ez = np.array([sim.get_array(center=cell_center,size=mp.Vector3(cell.x, cell.y, 0),component=mp.Ez)])
    hx = np.array([sim.get_array(center=cell_center,size=mp.Vector3(cell.x, cell.y, 0),component=mp.Hx)])
    hy = np.array([sim.get_array(center=cell_center,size=mp.Vector3(cell.x, cell.y, 0),component=mp.Hy)])
    hz = np.array([sim.get_array(center=cell_center,size=mp.Vector3(cell.x, cell.y, 0),component=mp.Hz)])
    sim_handler.update_fields(ex, ey, ez, hx, hy, hz)

sim.run(mp.at_every(1, updateField), until=260)


sim_handler.save_data(cell.x, cell.y)

# Finds the S parameters
norm_mode_coeff = sim.get_eigenmode_coefficients(mode_monitor_1, [1], eig_parity=mode_parity).alpha[0, :, 0]
port1_coeff = sim.get_eigenmode_coefficients(mode_monitor_1, [1], eig_parity=mode_parity).alpha[0, :, 1] / norm_mode_coeff
port2_coeff = sim.get_eigenmode_coefficients(mode_monitor_2, [1], eig_parity=mode_parity).alpha[0, :, 1] / norm_mode_coeff
port3_coeff = sim.get_eigenmode_coefficients(mode_monitor_3, [1], eig_parity=mode_parity).alpha[0, :, 0] / norm_mode_coeff
port4_coeff = sim.get_eigenmode_coefficients(mode_monitor_4, [1], eig_parity=mode_parity).alpha[0, :, 0] / norm_mode_coeff

np.savez('updated/s_params.npz', wl=wl, s11=port1_coeff, s21=port2_coeff, s31=port3_coeff, s41=port4_coeff)```

[theogdoctorg]

Hi felme1,

According to the FAQ here, you can roughly estimate the memory requirements by 96sxsyszr^3 Bytes of memory, so plugging in 963310.56pow(res,3)*(1GB/1e-9B) ~= 25GB of memory.

However, I suspect the main cause of the high memory usage is not Meep but instead your "SimulationHandler" class. It appears that every dt = 1, "updateField" takes a slice of the cell and pulling 6 values, so each slice 8*(33x50)(10.550) ~5.2 million values or about 42 MB per timestep. It then appends the 6 slices to 6 running lists, which you are holding in memory via the numpy arrays, meaning by the end of the 260 timetsteps there will be an extra ~11 GB of data in memory. I'm not sure where the very large (~TB?) level of memory usage you saw may be coming from, but Python list appends are done "over-allocation" if the number of list elements exceeds certain sizes, which may be account for some extra. Since it doesn't look like this SimulationHandler logging is being used to calculate any further, I would suggest either removing it and relying on your S-parameter calculation, or streaming the data to disk directly instead of storing in RAM.

Does not seem like a Meep issue.

Hope this helps!

[felme1]

Hi theogdoctorg,

thank you for the quick reply!
I think I have found at leas part of the problem. I was running the code using mpi, to speed up the process, however upon closer inspection It seems like this was the reason so much storage was used.
I split the task among 14 cores, and according to the system manager all of them used up something like 180GB of ram. (That was without saving any field data at all, but using dispersive materials.) Currently I am running the same exact code with just a single process. This also requires 180GB storage, but only once. It is also definitely slower, but no slower than twice as long as with mpi on 14 cores.

However I still have some questions:

1. Now I understand 180GB is still quite a lot, but could this be because I am using dispersive materials? I also read in the meep docs (https://meep.readthedocs.io/en/latest/FAQ/#what-is-an-estimate-of-the-memory-requirements-of-a-simulation-given-only-its-cell-size-and-resolution )that the pml layers, or in this case the absorber layers, can also cause larger storage requirements.

2. In the meep docs (https://meep.readthedocs.io/en/latest/Parallel_Meep/#using-parallel-meep) it is said, that mpi can be used for simulations that are to computationally expensive to be run by just one process or to large for the storage of a single machine, without making any changes to the code. However, as I just pointed out, when I run the script using mpi, like this: mpirun -np 14 python coupler.py, the storage requirements seem to be multiplied by the number of processes and I also dont see a 14 fold speed up in computational time. Am I missing something here? Is my simulation for some reason not suited to be run using mpi? The machine I am working with has a total of 24 cores, so it is not like I have to many processes for to little cores.

Thank you in advance!

[stevengj]

Dispersive materials definitely take more memory and computation time. It depends on how many Lorentz–Drude terms you are using — I would suggest fitting a minimal number to your bandwidth of interest rather than using the materials library (which might use a much larger number to fit a larger bandwidth).

However, as I just pointed out, when I run the script using mpi, like this: mpirun -np 14 python coupler.py, the storage requirements seem to be multiplied by the number of processes and I also dont see a 14 fold speed up in computational time.

Are you using an MPI version of Meep? Check that meep.count_processors() is 14.

(Note that even for a parallel version of Meep you won't get linear speedup — with all parallel programs, there is eventually a diminishing return as you add more processes, depending on the computation time.)