| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| arcca:raven [2012/12/05 14:48] – michael.puerrer@LIGO.ORG | arcca:raven [2012/12/20 13:11] (current) – [Grid tools] stephen.fairhurst@LIGO.ORG |
|---|
| * pkg-config: This is available by default on Raven (as /usr/bin/pkg-config). The installed version is 0.23, which seems to suffice | * pkg-config: This is available by default on Raven (as /usr/bin/pkg-config). The installed version is 0.23, which seems to suffice |
| * gsl: This is available as a module. So run <code bash> module load gsl</code> to pick this up. Current version is 1.15, which is more recent than version 1.12 listed on the DASWG page. | * gsl: This is available as a module. So run <code bash> module load gsl</code> to pick this up. Current version is 1.15, which is more recent than version 1.12 listed on the DASWG page. |
| * git: This is required to check out the LALSuite software stack. It **should** be available as a module, but isn't working yet. To work around the issue, I've installed version 1.8.1 in /home/spxsf2/opt/lscsoft/non-lsc. | * git: This is required to check out the LALSuite software stack. It is available in /usr/bin (version 1.7.1), a newer version (1.7.9) is available with <code bash> module load git</code> |
| * fftw: This **should** be available as a module, but isn't working yet! I have installed it at /home/spxsf2/opt/lscsoft/. **Note**: the instructions call for fftw to be built twice -- float and double. Not sure if this is still relevant and/or can be set up on Raven by default. | * fftw: This **should** be available as a module, but isn't working yet! I have installed it at /home/spxsf2/opt/lscsoft/. **Note**: the instructions call for fftw to be built twice -- float and double. Not sure if this is still relevant and/or can be set up on Raven by default. |
| * libframe/metaio/glue: This is LIGO specific software. It is not available as a module, but in the longer term, we should see if we can get it installed. For now, I downloaded the "stable" version from [[http://www.lsc-group.phys.uwm.edu/daswg/download/software/source/ | here]] and have installed (following the instructions [[https://www.lsc-group.phys.uwm.edu/daswg/docs/howto/lscsoft-install.html | here]]) at /home/spxsf2/opt/lscsoft/. | * libframe/metaio/glue: This is LIGO specific software. It is not available as a module, but in the longer term, we should see if we can get it installed. For now, I downloaded the "stable" version from [[http://www.lsc-group.phys.uwm.edu/daswg/download/software/source/ | here]] and have installed (following the instructions [[https://www.lsc-group.phys.uwm.edu/daswg/docs/howto/lscsoft-install.html | here]]) at /home/spxsf2/opt/lscsoft/. |
| ===== Grid tools ===== | ===== Grid tools ===== |
| |
| I'm trying to install the LIGO Data Grid Client, instructions [[https://www.lsc-group.phys.uwm.edu/lscdatagrid/doc/installclient-source.html | here]]. It fails with the following issue: | Instructions to install the LIGO Data Grid Client from source are taken from [[https://www.lsc-group.phys.uwm.edu/lscdatagrid/doc/installclient-source.html | here]]. |
| |
| <code bash>/usr/bin/gcc -DHAVE_CONFIG_H -I. -I. -I.. -I/home/spxsf2/opt/lscsoft/non-lsc/globus/include/globus -I/home/spxsf2/opt/lscsoft/non-lsc/globus/include/globus/gcc64dbg -g -m64 -Wall -DGLOBUS_PREFIX=\"/home/spxsf2/opt/lscsoft/non-lsc/globus\" -DGLOBUS_LIBDIR=\"/home/spxsf2/opt/lscsoft/non-lsc/globus/lib64\" -c globus_thread.c -fPIC -DPIC -o .libs/globus_thread.o | <code> |
| globus_thread.c:38:18: error: ltdl.h: No such file or directory | wget http://www.globus.org/ftppub/gt5/5.2/5.2.0/installers/src/gt5.2.0-all-source-installer.tar.gz |
| globus_thread.c: In function 'globus_i_thread_pre_activate': | tar xf gt5.2.0-all-source-installer.tar.gz |
| globus_thread.c:162: error: 'lt_dlhandle' undeclared (first use in this function) | mkdir gt5.2.0-all |
| globus_thread.c:162: error: (Each undeclared identifier is reported only once | export GLOBUS_LOCATION=~/gt5.2.0-all/ |
| globus_thread.c:162: error: for each function it appears in.) | export PATH=/bin:/usr/bin; |
| globus_thread.c:162: error: expected ';' before 'impl_lib' | export FLAVOUR=gcc64dbg |
| globus_thread.c:172: warning: implicit declaration of function 'lt_dlinit' | |
| globus_thread.c:173: warning: implicit declaration of function 'lt_dladdsearchdir' | cd gt5.2.0-all-source-installer |
| globus_thread.c:196: error: 'impl_lib' undeclared (first use in this function) | ./configure --prefix=$GLOBUS_LOCATION --with-flavor=$FLAVOUR |
| globus_thread.c:196: warning: implicit declaration of function 'lt_dlopenext' | make gsi-openssh |
| globus_thread.c:199: warning: implicit declaration of function 'lt_dlerror' | make postinstall |
| globus_thread.c:199: warning: format '%s' expects type 'char *', but argument 3 has type 'int' | . $GLOBUS_LOCATION/etc/globus-user-env.sh |
| globus_thread.c:204: warning: implicit declaration of function 'lt_dlsym' | |
| globus_thread.c:207: warning: format '%s' expects type 'char *', but argument 2 has type 'int' | |
| globus_thread.c: In function 'globus_thread_exit': | |
| globus_thread.c:1383: warning: cast from pointer to integer of different size | |
| make[3]: *** [globus_thread.lo] Error 1 | |
| make[3]: Leaving directory `/home/spxsf2/opt/src/gt5.2.3-all-source-installer/source-trees/common/source/library' | |
| make[2]: *** [all-recursive] Error 1 | |
| make[2]: Leaving directory `/home/spxsf2/opt/src/gt5.2.3-all-source-installer/source-trees/common/source' | |
| make[1]: *** [all] Error 2 | |
| make[1]: Leaving directory `/home/spxsf2/opt/src/gt5.2.3-all-source-installer/source-trees/common/source' | |
| make: *** [globus_common-make] Error 2 | |
| </code> | </code> |
| |
| Any suggestions?? | The VDT Certificate Bundle can be installed using the instructions from [[https://www.lsc-group.phys.uwm.edu/lscdatagrid/doc/installclient-source.html | the same page]]. (Note the link to the archive is out of date). I was also found necessary to update the certificates. |
| |
| | <code> |
| | wget http://software.grid.iu.edu/pacman/cadist/1.32/osg-certificates-1.32.tar.gz |
| | tar xf osg-certificates-1.32.tar.gz -C $GLOBUS_LOCATION/share |
| | globus-update-certificate-dir |
| | </code> |
| | |
| | Now copy your Grid certificates into the ''.globus'' folder in your home directory and make sure the permissions are correct. |
| | |
| | <code> |
| | chmod 600 ~/.globus/usercert.pem |
| | chmod 400 ~/.globus/userkey.pem |
| | </code> |
| | |
| | To source the install, you need: |
| | <code> |
| | . ~spxph/gt5.2.0-all/etc/globus-user-env.sh |
| | </code> |
| |
| ===== Data ===== | ===== Data ===== |
| module load bullxmpi/bullxmpi-1.1.17.1 | module load bullxmpi/bullxmpi-1.1.17.1 |
| </code> | </code> |
| | * Use a standard MyConfig for bam -- you don't need to point it to specific mpi libraries; just use mpicc as the compiler. |
| * Recompile bam: | * (Re)compile bam: |
| <code bash>make clean && make -j 8 bam</code> | <code bash>make clean && make -j 8 bam</code> |
| |
| |
| #PBS -q workq | #PBS -q workq |
| #PBS -l select=16:ncpus=8:mpiprocs=8 | #PBS -l select=8:ncpus=16:mpiprocs=16 |
| #PBS -l place=scatter:excl | #PBS -l place=scatter:excl |
| #PBS -l walltime=1:00:00 | #PBS -l walltime=1:00:00 |
| |
| #PROJECT=PR37 | #PROJECT=PR37 |
| | |
| mpirun -np 128 /home/spxmp/MachineConfig/bam/exe/bam /home/spxmp/MachineConfig/ARCCA/R6_PN_64.par | pardir=/home/spxmp/MachineConfig/ARCCA/ |
| | parfile=R6_PN_64.par |
| | bamexe=/home/spxmp/MachineConfig/bam/exe/bam |
| | |
| | cd /scratch/spxmp |
| | cp $pardir/$parfile . |
| | |
| | mpirun -np 128 $bamexe ./$parfile |
| </code> | </code> |
| | |
| | * This pbs-script copies the parameter file over to the lustre filesystem in /scratch/$USER. This is where you should run, i.e. *not* in your home directory. |
| | * Don't forget to mirror the data with rsync to minion, since files will get deleted after some time. |
| | * At the moment we are using queue 'workq'. There is also a dedicated 'ligo' queue and we should maybe use that. |
| | |
