Shell script for checking status of PSU RCC clusters (thank, Robert)

Leave a reply

#!/bin/sh

#Script for checking status of PSU RCC clusters all at once

#Set this to your user id. If you don’t have your ssh-key set up you are going to have to
#enter your password a lot, so do that first.
user=rcm242

echo “”
#You can add or remove cluster names here, include everything except .psu.edu
for cluster in cyberstar lionxv.rcc lionxi.rcc lionxj.rcc lionxg.rcc lionxf.rcc lionxh.rcc ; do
ssh -o LogLevel=quiet $user@$cluster.psu.edu ‘qstat -B; echo “”; showq | grep -A 2 “processors in use by local”‘
echo “”
done

MultyVac

Leave a reply

Another platform to facilitate rapid development of cloud applications is now becoming available.
Check out MultiVac.
I haven’t tried it out myself yet, but please feel to use MultyVac instead of Domino for Lab 7.
While Domino is limited to one node per job, my understanding is that MultyVac allows (or maybe will eventually allow) for jobs that instantiate a cluster.
If anyone write some scripts or functions that make it easier to use Julia with MultyVac, please share!

Examples of using @parallel for loop & parallel map with a distributed array

Leave a reply

For the parallel for loop, result = \sum_{i=1}^{10} i would be coded like
result = @parallel (+) for i in 1:10
i
end

To apply map in parallel on a distributed array, the syntax looks like:
addprocs(4)
@everywhere func(i, j; frac= 0.0) = i*100+j+frac
n = 4
m = 5
arr_local = [ (i,j) for i in 1:n, j in 1:m ]
arr_dist = distribute(arr_local)
result_dist = map(tuple -> func(tuple[1], tuple[2],frac=0.5), idx)
result_local = convert(Array{Float64},result_dist)

Computer Clusters at PSU

Leave a reply

https://rcc.its.psu.edu/resources/hpc/

There’s also a new lion-xv cluster that’s not yet in the table.

Students in Astro 585 should all have an account on Hammer, Lion-XF, Lion-XG, Lion-XV, Lion-XH, Lion-XI, and Lion-XJ.
Your account uses your Penn State Access Account userid and password.
Information on using and accessing these systems can be found at:http://rcc.its.psu.edu/user_guides/systems

RCC staff members are available to meet with you in person to go over compute, data and software resources that might be needed for your research and answer questions. In addition RCC offers several training seminars and workshops available at http://its.psu.edu/training

If you still have questions after you have gone over the detailed information available on RCC website above, please feel free to request a meeting with an RCC staff member by sending an e-mail to helpdesk@rcc.its.psu.edu

Philosophical musing on Astronomy, Computer Science, Statistics and you

Leave a reply

A class member brought my attention to the following article:
The True Bottleneck of Modern Scientific Computing in Astronomy.
There’s some interesting ideas to reflect on.

I recently ran across another interesting perspective on some of the challenges and opportunities for science and scientists who have multidisciplinary training.

Feel free to post links to other articles or blogs about some of these issues and to share your thoughts in the comments.

We should plan talk these types of issues during class some time. I’ll think about where best to fit it in.

How to tips for Garbage Collection in Julia

Leave a reply
If you have a large array A and want Julia to clear out that memory, the easiest way to do that in Julia is

A = 0  # set A to point to a single integer
gc()

If you have a performance sensitive part of your code and _don’t want the garbage collector to run during a certain time, you can call

gc_disable()

When you’re ready to turn it back on, you can run

gc_enable()

The following macro may be helpful for suspending garbage collection for one function call

macro nogc(ex)
quote
local val
try
gc_disable()
val = $(esc(ex))
finally
gc_enable()
end
val
end
end

Then you can ask Julia to a function without invoking the garbage collector during the run.
@nogc my_function()

In some cases (e.g., a loo with many operations on small arrays) this can provide a substantial performance boost. Try it!