<< InhomCosmIII

Inhomogeneous Cosmologies III Software notes

See also:

Einstein Toolkit tutorial

Workshop notes - ET

Download FLRWSolver and add it to the ET

  • Clone the public FLRWSolver version from bitbucket using: git clone https://hayleymac@bitbucket.org/hayleymac/flrwsolver-public-2.git
  • Rename the folder flrwsolver, and move to Cactus/arrangements/EinsteinInitialData/
  • Edit thornlists/einsteintoolkit.th in your top Cactus directory
  • Find EinsteinInitialData
  • Add the line: EinsteinInitialData/FLRWSolver beneath other thorns
  • You may safely comment out the line Llama/WaveExtractL since it is heavy (RAM/cpu time) to compile and not needed.
  • You may also safely comment out EinsteinInitialData/NoExcision, or any of the Meudon_BH thorns (if you run into errors)
  • Compile: ./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist=thornlists/einsteintoolkit.th --optionlist=myoptionlist.cfg
    • where myoptionlist.cfg is the appropriate optionlist for your machine -- look in simfactory/mdb/optionlists for a list
  • (You will probably first be asked to use sim setup ; you should create a directory where you want to store simulations, and then run ./simfactory/bin/sim setup ; for local usage, you can accept the defaults for the "key" values.)
  • Run: ./simfactory/bin/sim submit FLRW1 --parfile=path/to/FLRW.par --procs=1 --walltime=0:30:0
  • OR: navigate to your simulations/ directory and simply run: ./cactus_sim FLRW.par (where cactus_sim is the executable, located in Cactus/exe/. Create a softlink in your simulations directory)
  • EDIT MONDAY NIGHT: There were some typo's in FLRW.par, FLRW_perturb.par, and the Python scripts you will use in the next part. Please re-pull from the FLRWSolver repository. This won't affect your compiled Cactus. Navigate to Cactus/arrangements/EinsteinInitialData/flrwsolver/ and type git pull (if you originally cloned the public repository, if not: just ask Hayley and she'll give you the updated files via USB/email).

Checking your solutions

  • You will also find a simple Python script exactsolns.py in the FLRWSolver repository to check how your simulation is working.
  • Check the script, and make sure the indicated parameters match your simulation
  • This will generate some PDF plots which you can see how well you match the FLRW solution

More interesting runs

  • We can repeat the above run using instead FLRW_perturb.par to simulate a simple single-mode perturbation to a background FLRW spacetime
  • Once this is done, the same python script exactsolns.py can be edited to instead compare solutions for the growth of linear perturbations (set FLRW=False)
  • Note that the perturbations may not match the exact solution very well -- this is simply because we are currently running a 10^3 grid, so increase the resolution (change dx,dy,dz in your parameter file) and you will see a better match!
  • We can use SPLASH to visualise our 3D HDF5 data output from Cactus.


  • Usually used for visualising particle distributions
  • Daniel has modified it to read in Cactus HDF5 data
  • We first need to modify the Cactus output to have one file (with all variables included) per timestep. To do this, run the Python script (also located in the FLRWSolver repository): python split_HDF5_per_iteration.py
  • This is written using Python2.7, and you will need to install h5py and numpy
  • Once you have run this, you will have some files named FLRW_perturb_it0.hdf5, FLRW_perturb_it10.hdf5... and so on.
  • These files can then be visualised using SPLASH. Download from: http://users.monash.edu.au/~dprice/splash/ and first compile as per instructions on the webpage
  • We then need to compile the Cactus capabilities separately: make SYSTEM=gfortran cactus
  • You should now be able to visualise your data with: csplash-hdf5 FLRW_perturb_it0.hdf5 and select from the list which variable you want to look at!
  • Enjoy!


  • avoid filling home directory: To avoid creating a huge (733 Mb) directory in your home directory for the simplified tutorial for new users, create your target directory and soft link it to ~/simulations either before you start or before you start running a simulation.

Successful runs of tutorial

Gevolution hackathon

  • gevolution:
    • https://github.com/gevolution-code/gevolution-1.1
    • ArXiv:1604.06065
    • makefile edit for Debian GNU/Linux (stretch):
      INCLUDE += -I../LATfield2 -I/usr/include/hdf5/serial -L/usr/lib/x86_64-linux-gnu/hdf5/serial
      INCLUDE += -I../LATfield2 -I/usr/include/hdf5/openmpi -L/usr/lib/x86_64-linux-gnu/hdf5/openmpi
    • LATfield2 hack: see this hdf5.h hack to daverio LATfield2
    • if you unload both package directories at the same directory level, then in gevolution-1.1/ you can do ln -s ../LATfield2 . so that the gevolution makefile can find the LATfield2 files
    • Create an output directory mkdir output/
    • example libraries you will need: sudo aptitude install libfftw3-dev openmpi-bin libopenmpi-dev libhdf5-openmpi-dev libhdf5-dev hdf5-helpers
    • You should be able to compile gevolution with make and run the example with
      (time mpirun -np 4 ./gevolution -n 2 -m 2 -s settings.ini) > gevolution.log 2>&1
      in order to get a nice log file - on Boud's 4-core laptop this took about 140 seconds of user time and created many output files in output/ .
  • Visualisation - SPLASH:


  • To test run on a machine with less than four cores, a hack to try is: mpirun -np 4 --map-by core:OVERSUBSCRIBE" ; you might get a segmentation fault at the end of the run, after output files are created


Ray-tracing: gyoto

-- BoudRoukema - 13 Sep 2018 + ...
  • gevolution phi P(k) example:
Topic attachments
I Attachment Action Size Date Who Comment
gevolution_phi_test.pngpng gevolution_phi_test.png manage 2.1 K 18 Sep 2018 - 15:23 BoudRoukema gevolution phi P(k) example
Topic revision: r20 - 07 Nov 2018, BoudRoukema
This site is powered by FoswikiCopyright © CC-BY-SA by the contributing authors. All material on this collaboration platform is copyrighted under CC-BY-SA by the contributing authors unless otherwise noted.
Ideas, requests, problems regarding Foswiki? Send feedback