MAIA bb96820c
Multiphysics at AIA
Loading...
Searching...
No Matches
Getting Started (AIA Users)

Information for AIA users

This section contains information specific for the compute environment of the AIA. As you are an AIA user, this quickstart will also inform you about some of the peculiarities of the IT infrastructure at the AIA and some best practices.

Note
Before installing m-AIA at the AIA, you need to get an user account (including a password) at the institute by contacting Matthias Meinke (Room 206). It is also important that you know a bit about the IT infrastructure at the AIA. There are two directories:
  • home: This directory has very limited space (in particular it is very small for students). The main reason for the limited storage, is that a backup of all home directories is performed every night to the computing center of the RWTH. Tip: Use the command cd ~ to get to your home directory.
  • scratch: This directory is in a system with a larger disk space, which is shared with other users. It is very easy to accidentally generate a lot of output data with m-AIA and completely fill up the scratch space. In that case, other users will not be very happy with you, since their own simulations won't be able to run any more, which for you means baking cakes, lots of them, as a compensation. In case you really need to write a lot of output to the scratch space, please check the free space before doing so with the unix command df. If you don't have a scratch directory, Miro Gondrum (Room 001) will be happy to provide one for you. To ease working with both your home and scratch directory, it is usually a good idea to create a link in your home directory to your scratch space with the following command cd ~/; ln -s /aia/<raid device>/scratch/<your user name> ./. You can then use the command cd ~/scratch to change get to your personal scratch directory. You will get the raid device number from Miro Gondrum (Room 001). Alternatively you could also set an an environment variable called, e.g. SCRATCH in your .bash_profile such as export SCRATCH=/aia/<raid device>/scratch/<your user name> and use the command cd $SCRATCH to change into your scratch directory.

For the reasons mentioned above, this is what you should store where:

  • home directory: In general, this is a location for all files generated by a text editor such as:

    • source code of your own m-AIA branch(es)
    • text documents, e.g. of your thesis
    • parameter or property files
    • python or matlab scripts for the postprocessing of simulation data

    Do not store files containing simulation results in your home directory!

  • scratch directory: In general, this is the location for all results generated by a simulation run such as:

    • restart files
    • snapshots needed for the postprocessing of results

    Do not store data on your scratch space, which cannot be reproduced by a simulation run!

Before generating new data on any file server, always make sure that the available free space is sufficiently large by using the unix command df Keeping the information above in mind, you can now start installing and using m-AIA by first cloning the git repository (repo) of m-AIA, second configuring m-AIA, third compiling m-AIA and fourth running your first simulation, e.g. a testcase or tutorial.

  1. Clone the repo of m-AIA: For this step git is used.

    Open the terminal on your local Linux machine:

    Ctrl + Alt + T

    Connect to a frontend (fe) of the AIA cluster, e.g., fe1:

    ssh fe1

    Tip: Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

    If you want to isolate your m-AIA simulation project(s), create a folder (yourFolder) where you want to clone the repository on your scratch and change the directory to this folder:

    cd ~/scratch/
    mkdir yourFolder
    cd yourFolder

    Clone the m-AIA repo:

    git clone git@git.rwth-aachen.de:aia/MAIA/Solver.git

    Change the directory to the downloaded Solver folder and switch to the desired branch you want to work with:

    cd Solver
    git checkout yourBranch

    Tip: If you want to work with the main/master branch, you do not need to switch the branch.
    Tip: Use git fetch and then git status to check the current status.


  2. Configure m-AIA: This is only guaranteed to work on the frontend like fe1 and cluster nodes, so ssh fe1 if not already.

    Run the configure.py file in the top directory (Solver folder):

    ./configure.py 1 2

    Tip: 1 2 for gnu production mode. You can also use ./configure.py gnu production.
    Tip: You can also use the default setting with ./configure.py ? ?.

    This process should take about 1 minute. The print message on the console should look like this:

    configure.py: current host: AIA
    configure.py: selected compiler: GNU
    configure.py: selected build type: production
    configure.py: selected parallelism type: MPI+OpenMP
    configure.py: compile with HDF5 support in BigData: enabled
    configure.py: update git submodules (skip this step with --disable-updateGitSubmodules)
    Submodule 'include/Eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'include/Eigen'
    Submodule 'include/cantera' (https://github.com/Cantera/cantera.git) registered for path 'include/cantera'
    Cloning into '... /Solver/include/Eigen'...
    Cloning into '... /Solver/include/cantera'...
    Submodule path 'include/Eigen': checked out 'cd80e04ab77bec92c63a4cce1e089262334d23a9'
    Submodule path 'include/cantera': checked out '9cee0525d26b6030cc30c469c9c75332d836eb8e'
    configure.py: configuring @maia
    configure.py: @maia is ready for make!


  3. Compile m-AIA: This is only guaranteed to work on cluster nodes, so ssh fe1 if not already.

    Check for available resources on the connected frontend:

    shosts

    The print message on the console should look like this:

    node state load ncpu free mem partition user jobid time limit
    ============================================================================================
    aia319 idle 0.63 1 50310 quadro6000
    aia323 idle 0.96 1 17858 quadro6000
    aia324 idle 0.48 1 10930 quadro6000
    aia320 idle 0.45 1 11075 rtx2080ti
    aia321 idle 0.14 1 4314 rtx2080ti
    aia322 idle 0.13 1 12719 rtx2080ti
    aia260 idle 0.20 2 184056 forty
    aia261 idle 0.18 2 190726 forty
    aia262 comp 24.63 2 173362 forty
    aia263 idle 1.28 2 54008 forty
    aia292 alloc 0.21 2 23717 twelve moritzw 2030466 3:58:13 4:00:00
    aia293 idle 0.00 2 22668 twelve* yourName JobID remaining time total time
    aia294 idle 6.98 2 22612 twelve*
    aia295 idle 3.66 2 19229 twelve*
    aia296 idle 6.81 2 18287 twelve*
    aia297 idle 2.75 2 19037 twelve*
    aia298 idle 4.21 2 21130 twelve*
    aia299 idle 5.05 2 19929 twelve*
    aia300 idle 4.56 2 21226 twelve*
    aia301 idle 4.24 2 22471 twelve*
    aia302 idle 4.33 2 19564 twelve*
    aia303 idle 5.15 2 20031 twelve*
    aia304 idle 5.84 2 20529 twelve*
    aia305 idle 5.91 2 19379 twelve*
    aia306 idle 3.64 2 16011 twelve*
    aia307 idle 11.15 2 12997 twelve*
    aia280 idle 0.00 2 127044 twentyeight
    aia281 idle 0.34 2 60457 twentyeight
    aia314 idle 0.00 2 119543 twentyeight
    aia316 idle 0.00 2 123808 twentyeight
    aia317 idle 0.00 2 113718 twentyeight
    aia318 down* 0.06 2 181061 twentyeight newSlurmVersion
    aia308 down* 24.02 2 119166 twentyfour unkownYet
    aia309 idle 0.00 2 122173 twentyfour
    aia310 idle 21.21 2 103879 twentyfour
    aia311 alloc 0.10 2 126183 twentyfour yu 2031210 1:30:56 4:00:00
    aia312 alloc 24.05 2 74333 twentyfour shangyu 2031218 1:12:15 4:00:00
    aia313 idle 0.01 2 128119 twentyfour

    Allocate a cluster node (interactive session) on the AIA cluster with si queue noNodes hh:mm:ss to allocate a cluster node for a certain amount of time hh:mm:ss like 00:15:00 for 15min, where queue is the number of jobs/processors of the node you want to use, e.g. 12, noNodes is the number of nodes you want to allocate, e.g. 1.

    si 12 1 00:15:00

    Run the Makefile in the top directory (Solver folder) to compile m-AIA in parallel using noCores processors, here 12:

    make -j 12

    This process can take 10 to 15 minutes. The print message on the console should look like this:

    Scanning dependencies of target maia
    [ 0%] Building CXX object src/CMakeFiles/maia.dir/GRID/cartesiangridgenpar_inst_2d.cpp.o
    [ 2%] Building CXX object src/CMakeFiles/maia.dir/GRID/cartesiangrid_inst_2d.cpp.o
    [ 3%] Building CXX object src/CMakeFiles/maia.dir/GRID/cartesiangridgenpar_inst_3d.cpp.o
    [ 3%] Building CXX object src/CMakeFiles/maia.dir/GRID/cartesiangrid_inst_3d.cpp.o
    [ 3%] Building CXX object src/CMakeFiles/maia.dir/GRID/cartesiangridgencell.cpp.o
    [ 4%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_2d_rans_fs.cpp.o
    [ 4%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_2d_rans_sa.cpp.o
    [ 4%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_3d_ns.cpp.o
    [ 5%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_2d_ns.cpp.o
    [ 5%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_2d_detchem.cpp.o
    [ 7%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_2d_rans_komega.cpp.o
    [ 7%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_3d_rans_sa.cpp.o
    [ 7%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_3d_rans_fs.cpp.o
    [ 8%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_3d_rans_komega.cpp.o
    [ 9%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_3d_eegas.cpp.o
    [ 9%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansolverxd_inst_3d_detchem.cpp.o
    [ 10%] Building CXX object src/CMakeFiles/maia.dir/FV/fvcartesiansyseqnns.cpp.o
    ...
    [ 90%] Building CXX object src/CMakeFiles/maia.dir/DG/dgcartesiansolver.cpp.o
    [ 90%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/couplerfvmultilevel.cpp.o
    [ 91%] Building CXX object src/CMakeFiles/maia.dir/GEOM/geometryroot.cpp.o
    [ 92%] Building CXX object src/CMakeFiles/maia.dir/FV/fvstg.cpp.o
    [ 92%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/lslb.cpp.o
    [ 93%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/couplerlbfv.cpp.o
    [ 93%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/couplerlbfveemultiphase.cpp.o
    [ 94%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/couplerlblb.cpp.o
    [ 95%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/lbrb.cpp.o
    [ 95%] Building CXX object src/CMakeFiles/maia.dir/ACA/acasolver.cpp.o
    [ 96%] Building CXX object src/CMakeFiles/maia.dir/RB/rigidbodies.cpp.o
    [ 96%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/lslbsurface.cpp.o
    [ 97%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/lblpt.cpp.o
    [ 98%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/lbdgape.cpp.o
    [ 98%] Building CXX object src/CMakeFiles/maia.dir/COUPLER/couplingdgape.cpp.o
    [ 99%] Building CXX object src/CMakeFiles/maia.dir/maia.cpp.o
    [100%] Linking CXX executable maia
    [100%] Built target maia

    As a test, if m-AIA is compiled open m-AIA's help:

    cd src
    maia -h


  4. Run your first m-AIA simulation: This is only guaranteed to work on the frontend like fe1 and cluster nodes.

    Check and allocate a cluster node on the AIA cluster with noCores cores, e.g. 12, if not already:

    shosts
    si 12 1 00:30:00

    Tip: By using shosts again your username should be listed for one of the nodes of the AIA cluster inlcuding the remaining time for which you have allocated this node.

    Go to your simulation project (e.g. a testcase or tutorial) and create a link to the compiled m-AIA executable of step 3:

    cd simulationProject
    ln -s ~/scratch/Solver/src/maia maia

    As a test, check if the link is successfully created by openin m-AIA's help:

    ll
    maia -h

    Run the simulation in parallel. The number of cores noCores, here 12, should match with the number of cores allocated:

    mpirun -np 12 maia properties.toml

    Tip: The property file contains resp. specifies the most important solver settings (properties) to run the simulation.

    Tip: Run scancel JobID to cancel a running job, where the JobID is identified via the table displayed by using shosts.


Note
For further information how to use certain solver parts of m-AIA and performing your first pre- and postprocessing, please refer to the tutorials.

m-AIA testcases

Now that the code has been compiled it can be used to simulate testcases. Some ready-to-use testcases are found in the subfoldera of the testcase repository here http://svn/maia/testcases/. When being in the AIA network you can use your browser to open the testcases and to have a look at them or use the terminal to check the testcase out (=copy) with the following command.

cd ~/scratch
svn co http://svn/maia/testcases/
cd yourTestcaseYouWantToExplore

Now create a symbolic link (command ln -s) to your m-AIA executable maia (compiled code in yourDirectory = ~/scratch/yourFolder) in your chosen testcase folder to avoid unnecessary copies of the executable.

ln -s ~/scratch/yourFolder/Solver/src/maia maia

Once the m-AIA executable has been created and linked, the simulation can be started as follows, where properties_run.toml is an ASCII format containing the settings to run the m-AIA-based simulation.

mpirun -np 12 maia properties_run.toml
Warning
Don't start simulations on the login nodes fe1/fe2/fe3 as it will consume most of the resources and slow down the node for all users. For computations you have two options:
  • Get an interactive session (command si) on the AIA cluster before starting the simulation. This means that for a limited time, 30 minutes, you will have exclusive access to one of the computation nodes with for example 12 cores on the AIA cluster.
    ssh fe1
    shosts
    si 12 1 00:30:00
    Tip: The command shosts is used to get an overview over all cluster nodes and their current status.
  • Use a batch file to send the job to the cluster. Once the resources are free the job will start execution and run mAIA. The advantage is that it will run independently from your current session so you can logout and let it do the work overnight

m-AIA tools

You can find m-AIA tools, which are useful, e.g., for pre- and postprocessing, here: https://git.rwth-aachen.de/aia/MAIA/tools

SSH tips

To use ssh to login into a remote node without the need to provide a password, you have to generate a key file first with ssh-keygen and then add the contents of your key file to your authorized_keys files in your .ssh directory:

Open the terminal

Ctrl + Alt + T

Generate a key in your home directory

ssh-keygen -t rsa

Go to your .ssh directory

cd ~/.ssh

Append the contents of the key file id_rsa.pub to authorized_keys

cat id_rsa.pub >> authorized_keys