{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Downloading Simulation Data\n",
    "\n",
    "## Simulation Access with SimIM\n",
    "\n",
    "**SimIM** draws galaxy (and dark matter halo) properties from cosmological simulations. The first step to using **SimIM** is therefore to download the necessary subhalo catalogs. Currently, **SimIM** supports subhalo catalogs from [Illustris](https://www.illustris-project.org/) and [IllustrisTNG](https://www.tng-project.org/), along with the [UniverseMachine](https://www.peterbehroozi.com/data.html) halo catalogs for the BolshoiPlanck and MultiDark simulations. Details on each simulation can be found on the linked web pages and associated publications. Any work using **SimIM** should separately acknowledge and reference the simulation used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading a Simulation\n",
    "\n",
    "In this example and throughout the **SimIM** documentation we use TNG100-3-Dark simulation. This is a simulation box with 100 Mpc sides, a dark matter particle mass of $4\\times10^8$ $M_\\odot$, and no Baryonic physics. We use this simulation because the total data volume is relatively small (~3 GB compared to ~1 TB for the full physics, full resolution TNG300-1 simulation). For most scientific applications it is advisable to utilize a higher resolution simulation. However the smaller, low-resolution TNG100-3 can be downloaded, formatted, and loaded much more quickly, making it useful for demonstration and testing purposes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can start by loading the ``simim.siminterface`` module, which is used for downloading and interacting with simulation halo catalogs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import simim.siminterface as sim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If this is your first time using **SimIM** on a machine, you may also need to run the following lines and follow the prompts to set a path in which to store downloaded simulation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from simim import setupsimim\n",
    "setupsimim()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can proceed to download the subhalo catalogs for our prefered simulation. Note that this is a large quantity of data and can take some time depending on your internet connection. This initial download is in the raw format provided by the original halo catalogs, we will translate it to SimIM's format in a second step.\n",
    "\n",
    "For Illustris and IllustrisTNG data, you will need an API key from the Illustris/TNG project, which can be obtained [here](https://www.tng-project.org/users/register/). For UniverseMachine catalogs no key is required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download the data - set the redownload parameter to True\n",
    "# if you want to overwrite snapshots already saved on your \n",
    "# machine.\n",
    "\n",
    "api_key = '[fill in your API key here]' # Replace this with a string containing key\n",
    "cat = sim.IllustrisCatalogs('TNG100-3-Dark',api_key)\n",
    "cat.download_meta()\n",
    "cat.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the simulation is downloaded we can reformat it to use with **SimIM**. This procedure reformats the data in a uniform format used by **SimIM** to interface with all supported simulations.\n",
    "\n",
    "Formatting is handled by the ``cat.format`` method. A few options are supported for formatting. Setting the ``basic`` parameter to ``True`` keeps only a minimal set of halo properties - position velocity, mass, radius, and a few others - which in the formatted catalog. This can be done to save space when other properties will not be needed.\n",
    "\n",
    "Setting the ``realtime_clean_raw`` parameter to ``True`` will delete the unformatted data (which is kept on the disk by default) as the **SimIM** formatted file is generated. This can save a significant amount of disk space when working with larger simulations, but will require re-downloading data if something goes wrong. We can then also delete the data later, once we've verified that the catalog has been successfully converted to the **SimIM** format, and this approach should be prefered when disk space is not a concern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat.format(remake=True,basic=False,realtime_clean_raw=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To remove the unformatted versions of the catalog, we can now call the ``cat.clean_raw`` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat.clean_raw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Partial Simulation Downloads\n",
    "\n",
    "It is also possible to save time and disk space by only downloading and formatting snapshots you need. For instance, if we only wanted the simulation from redshifts 0 to 2 (snapshots 33-99 for IllustrisTNG), then we could do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "snaps = np.arange(33,100)\n",
    "api_key = '[fill in your API key here]' # Replace this with a string containing key\n",
    "cat = sim.illustris.IllustrisCatalogs('TNG100-3-Dark',api_key,snaps=snaps)\n",
    "cat.download_meta(redownload=True)\n",
    "cat.download()\n",
    "cat.format(remake=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that some of the metadata saved with the formatted snapshots depends on the full list of snapshots to be included in the **SimIM** catalogs. Therefore the ``cat.download_meta``, ``cat.download`` and ``cat.format`` methods should all be run with the a  ``simim.siminterface.IllustrisCatalogs`` instance initialized with the same snapshot list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Simulations\n",
    "\n",
    "All simulations in the Illustris and IllustrisTNG can be downloaded and formatted using the ``simim.siminterface.IllustrisCatalogs`` class specifying the name of the desired simulation as the ``sim`` argument. The available simulations are ``Illustris-[N]`` and ``Illustris-[N]-Dark`` (where [N] is replaced with 1, 2, or 3), and ``TNG[S]-[N]`` and ``TNG[S]-[N]-Dark`` (where [S] is 50, 100, or 300 and [N] is again 1, 2, or 3).\n",
    "\n",
    "UniverseMachine catalogs for the BolshoiPlanck, MultiDark Planck 2, and Small MultiDark Placnk simulations. These can be accessed via ``simim.siminterface.UniversemachineCatalogs`` class and the names ``UniverseMachine-BolshoiPlanck``, ``UniverseMachine-MDPL2`` and ``UniverseMachine-SMDPL``."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also relatively straightforward to implement support for new simulations\n",
    "within the **SimIM** framework. Doing so requires adding new code to the \n",
    "**SimIM** package, which can be added by cloning the source code from the\n",
    "[SimIM project GitHub](https://github.com/rpkeenan/simim_public).\n",
    "The procedure for implementing new simulations is then described in the\n",
    "documentation for the ``simim.siminterface._rawsiminterface`` module."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "simim",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}