From: Paul H. Hargrove (PHHargrove_at_lbl_dot_gov)
Date: Mon Mar 22 2010 - 15:19:18 PDT
fengguang, As an author of the BLCR software, I cannot guess from your question what the problem might be. You say "the MPI program are not terminated as what happened in single machine", but you don't indicate if you used the "--term" argument in either the single machine or cluster case. One should not expect the program to terminate immediately in either case unless "--term" is passed (to cr_checkpoint or ompi-checkpoint, respectively). You say "ompi-restart also doesn't work on cluster", but you don't provide any information on how it is failing. Is there any error message from ompi-restart? Does the ompi-restart command exit, or does it hang? Unless OMPI is doing something special, I think that if the original MPI application is still running when you try to restart, then a restart failure is the expected behavior because BLCR is going to try to allocate the original PIDs. Since your problems sound specific to your use of MPI, I suggest that your best chance of getting a good answer would be to ask on the ompi-users mailing list. You can find out about the Open MPI mailing lists at www.open-mpi.org. -Paul fengguang tian wrote: > I set up a cluster of 18 nodes using Open MPI and BLCR library, and > the MPI program runs well on the clusters, > but how to checkpoint the MPI program on this clusters? > > what I have done is that: I run the program using mpirun in the shared > directory on the master node, and use > ompi-checkpoint command in another terminal on master node, it will > create an checkpoint file,but the MPI program > are not terminated as what happened in single machine. Also, the > ompi-restart also doesn't work on cluster. > > what should I do ? > > Cheers! > fengguang -- Paul H. Hargrove PHHargrove_at_lbl_dot_gov Future Technologies Group Tel: +1-510-495-2352 HPC Research Department Fax: +1-510-486-6900 Lawrence Berkeley National Laboratory