Downloading Simulation Data

Simulation Access with SimIM

SimIM draws galaxy (and dark matter halo) properties from cosmological simulations. The first step to using SimIM is therefore to download the necessary subhalo catalogs. Currently, SimIM supports subhalo catalogs from Illustris and IllustrisTNG, along with the UniverseMachine halo catalogs for the BolshoiPlanck and MultiDark simulations. Details on each simulation can be found on the linked web pages and associated publications. Any work using SimIM should separately acknowledge and reference the simulation used.

Downloading a Simulation

In this example and throughout the SimIM documentation we use TNG100-3-Dark simulation. This is a simulation box with 100 Mpc sides, a dark matter particle mass of \(4\times10^8\) \(M_\odot\), and no Baryonic physics. We use this simulation because the total data volume is relatively small (~3 GB compared to ~1 TB for the full physics, full resolution TNG300-1 simulation). For most scientific applications it is advisable to utilize a higher resolution simulation. However the smaller, low-resolution TNG100-3 can be downloaded, formatted, and loaded much more quickly, making it useful for demonstration and testing purposes.

We can start by loading the simim.siminterface module, which is used for downloading and interacting with simulation halo catalogs:

import simim.siminterface as sim

If this is your first time using SimIM on a machine, you may also need to run the following lines and follow the prompts to set a path in which to store downloaded simulation data.

from simim import setupsimim
setupsimim()

Now we can proceed to download the subhalo catalogs for our prefered simulation. Note that this is a large quantity of data and can take some time depending on your internet connection. This initial download is in the raw format provided by the original halo catalogs, we will translate it to SimIM’s format in a second step.

For Illustris and IllustrisTNG data, you will need an API key from the Illustris/TNG project, which can be obtained here. For UniverseMachine catalogs no key is required.

# download the data - set the redownload parameter to True
# if you want to overwrite snapshots already saved on your
# machine.

api_key = '[fill in your API key here]' # Replace this with a string containing key
cat = sim.IllustrisCatalogs('TNG100-3-Dark',api_key)
cat.download_meta()
cat.download()

Now that the simulation is downloaded we can reformat it to use with SimIM. This procedure reformats the data in a uniform format used by SimIM to interface with all supported simulations.

Formatting is handled by the cat.format method. A few options are supported for formatting. Setting the basic parameter to True keeps only a minimal set of halo properties - position velocity, mass, radius, and a few others - which in the formatted catalog. This can be done to save space when other properties will not be needed.

Setting the realtime_clean_raw parameter to True will delete the unformatted data (which is kept on the disk by default) as the SimIM formatted file is generated. This can save a significant amount of disk space when working with larger simulations, but will require re-downloading data if something goes wrong. We can then also delete the data later, once we’ve verified that the catalog has been successfully converted to the SimIM format, and this approach should be prefered when disk space is not a concern.

cat.format(remake=True,basic=False,realtime_clean_raw=False)

To remove the unformatted versions of the catalog, we can now call the cat.clean_raw method.

cat.clean_raw()

Partial Simulation Downloads

It is also possible to save time and disk space by only downloading and formatting snapshots you need. For instance, if we only wanted the simulation from redshifts 0 to 2 (snapshots 33-99 for IllustrisTNG), then we could do the following:

snaps = np.arange(33,100)
api_key = '[fill in your API key here]' # Replace this with a string containing key
cat = sim.illustris.IllustrisCatalogs('TNG100-3-Dark',api_key,snaps=snaps)
cat.download_meta(redownload=True)
cat.download()
cat.format(remake=True)

Note that some of the metadata saved with the formatted snapshots depends on the full list of snapshots to be included in the SimIM catalogs. Therefore the cat.download_meta, cat.download and cat.format methods should all be run with the a simim.siminterface.IllustrisCatalogs instance initialized with the same snapshot list.

Additional Simulations

All simulations in the Illustris and IllustrisTNG can be downloaded and formatted using the simim.siminterface.IllustrisCatalogs class specifying the name of the desired simulation as the sim argument. The available simulations are Illustris-[N] and Illustris-[N]-Dark (where [N] is replaced with 1, 2, or 3), and TNG[S]-[N] and TNG[S]-[N]-Dark (where [S] is 50, 100, or 300 and [N] is again 1, 2, or 3).

UniverseMachine catalogs for the BolshoiPlanck, MultiDark Planck 2, and Small MultiDark Placnk simulations. These can be accessed via simim.siminterface.UniversemachineCatalogs class and the names UniverseMachine-BolshoiPlanck, UniverseMachine-MDPL2 and UniverseMachine-SMDPL.

It is also relatively straightforward to implement support for new simulations within the SimIM framework. Doing so requires adding new code to the SimIM package, which can be added by cloning the source code from the SimIM project GitHub. The procedure for implementing new simulations is then described in the documentation for the simim.siminterface._rawsiminterface module.