Lab Notes

Important!

• If you are an undergrad, consider applying for the University's UROP funding, the IEEE Antennas and Propagation Society Pre-Doctoral Research Award, and the National Science Foundation Graduate Research Fellowship. See the COE Fellowships Office website.
• If you are a graduate student, consider applying for the IEEE Antennas and Propagation Society Doctoral Research Award, and the NSF Graduate Research Fellowship if you are a U.S. citizen. See the COE Fellowships Office website.
• Whenever you provide results to me, please take some extra time to think about how they are presented. For example:
  • Label the x- and y-axes in your plots.
  • Consider whether the information is better presented on a linear, semilog, or log-log scale.
  • Maybe provide overall plots and then zoom in on regions of interest.
  • Include a legend that labels what component is being graphed (East-West electric field, for example).
  • If you graph vs. time steps, include dt in the caption or somewhere so we can quickly calculate how many seconds have evolved over the simulation.
  Doing this will help me, and I think it will help all of us take a step back, think about what the results mean, and better compare them to real-world scenarios. It will also help us prepare to present our results in the best way to paper reviewers, etc.
• READ, READ, READ! Attend seminars, listen to talks at conferences, etc. Build a strong foundation of knowledge and be aware of the state of the art. This will help your research productivity immensely and will help you come up with your own ideas!
• Choose VERY secure passwords! An ideal password is long and has letters, punctuation, symbols, and numbers.
  • Whenever possible, use at least 14 characters.
  • The greater the variety of characters in your password, the better.
  • Use the entire keyboard, not just the letters and characters you use or see most often.
• Make back-up copies of your important codes and results (periodically and to archive)! It is very frustrating and a waste of time to try to recreate a code or result. Ask yourself, “If this computer were to disappear right now, would I be able to continue my work somewhere else without much interruption, or would I have to start over?”
• Put plenty of comments in your codes, both at the beginning to describe what the code does and throughout. If you come back to a code in a few months or years, it's easy to forget what is going on and what you were trying to do. Also, another student may continue your project after you graduate (leading to citations for your papers!), and they will need to understand your code.
• Keep a lab notebook! Include summaries after each of our meetings, progress, action items, sources of information, ideas, and the locations and names of codes you're currently using or working on, so that you can come back to the same version later if necessary.
• Store different versions of your codes as you work on them and as you finish new additions or changes. Sometimes a code can (seemingly) randomly “break,” meaning it won't work anymore and gives erroneous results in places where there are no errors, making it almost impossible to debug. In these cases, you will likely need to revert to a previous (working) version and redo the changes.
• The Center for High Performance Computing (CHPC) on campus periodically has presentations you may find useful, such as Introductory Linux, Introduction to MPI, and Intro to Programming with OpenMP. Check the CHPC website for the schedule (if you don't see any listed, contact them, because they always have some scheduled).
• If you are not a native English speaker and need to work on your English speaking, writing, or reading skills, it is important that you enroll in the U's English as a Second Language courses until you are proficient. Your tuition for these courses will be covered by the lab. More info here.
• To get help with your computer (an account on a lab computer, installing software, etc.), contact the engineering IT help desk at opers at eng dot utah dot edu.
• To see an example Ph.D. thesis, you can look at mine. Look at others as well, to get a feel for the general expectations for a Ph.D. thesis.
• As you progress through your M.S. or Ph.D. studies, you should become more independent, be able to suggest next steps for your research, and at some point start contributing your own ideas.
• If you need to order anything for the lab, go to the purchasing page of the ECE website and select Prof. Simpson to approve your purchase. We can order from Amazon, Office Depot, etc., as well as the University store.

Should I Pursue a Ph.D.?

A website to investigate different career options: myIDP (Science Careers)

Key factors to consider when deciding between the Ph.D. and the M.S.:

• Do you have the intellectual aptitude for the advanced concepts and courses involved specifically in the Ph.D. thesis work, and more generally in the broader related research area?
• Do you have the motivation and work ethic to persevere even when your research is not going smoothly?
• Do you want to work in a relatively unstructured manner, where you are not given much (if any) direction on how to define a problem or develop a solution approach?
• Do you want the option of teaching at the college level?

Note that a student's grade-point average in a Ph.D. program is not the figure of merit for employment. It is the quality and promise of the research, along with the letters of recommendation from the advisor, members of the Ph.D. committee, managers, etc. While a Ph.D. could be used simply as a credential to get a “better” job, or to improve one's standing within an existing company, it really should be seen as a means of extending the visibility of an individual engineer beyond any particular company to the technical world at large via conference presentations and journal publications. To this end, a Ph.D. can “rust,” i.e., decay, if the holder makes no effort to continue publishing.

What jobs are available for Ph.D.s? You can work in academia (as a research professor, if you do not want to teach), government (National Science Foundation, Air Force Research Labs, Office of Naval Research, one of the many National Laboratories, etc.), industry, and even go on to law school and become a patent attorney.

Become a Professor

• Have you considered becoming a professor? It is a very rewarding career. You can pursue exciting research of your choosing while also preparing the next generation of engineers.
• The back of IEEE Spectrum magazine is one of the best places to look for open positions.
• The U's Center for Teaching and Learning Excellence (CTLE) periodically offers a free Teaching Dossier Workshop to help you develop or sharpen your teaching philosophy statement.

Internships

There are many possibilities for internships.

For U.S. citizens, consider:

• AFRL Space Scholars Program
• Idaho National Labs
• Sandia National Labs
• NASA
• MITRE

For both U.S. and non-U.S. citizens, consider these (and other related opportunities):

• REMCOM
• COMSOL
• Intel Corp.
• Apple Inc.
• Optisys (in Utah)
• Ansys
• Lucid Motors
• Schlumberger Research
• Resonant
• Nanometrics
• Applied Materials, Inc.

Award and Fellowship Opportunities

See the COE Office of Fellowships website for more opportunities!

Undergraduate Students

• University of Utah Financial Aid and Scholarships
• U of U ECE and COE scholarships
• ECE Dept. top EE and top CE students, best paper and best presentation awards, and best clinic for senior design projects
• Apply to become a member of the U's Beehive Honor Society
• Check the U of U Office of Undergraduate Studies list of scholarships and the College of Engineering list of third-party scholarships
• Check the U of U Alumni Association scholarship list
• (For women; apply sophomore year by Feb.) University of Utah Women's Center, $1500 each
• Check the Utah Engineers Council website (look under forms)
• Conference student paper contests
• (For students with disabilities) Google Lime Scholarship and Lime Connect Fellowship Program
• (For U.S. citizens; apply during senior year to obtain support for graduate studies) National Science Foundation Graduate Research Fellowship
• Barry Goldwater Scholarship
• IEEE Antennas and Propagation Society Undergraduate Research Award (due April 1 and Nov. 1)
• IEEE Antennas and Propagation Society Design Contest
• IEEE Microwave Theory and Techniques Society Undergraduate Scholarships
• (For U.S. citizens) Science, Mathematics and Research for Transformation (SMART) Scholarship (Department of Defense)
• Computing Research Association Outstanding Undergraduate Researchers Award
• (For women) American Association of University Women Fellowships and Grants
• (For women) Fellowship, scholarship, and award opportunities listed with the Society of Women Engineers
• (For minorities) HSF/Marathon Oil Corporation College Scholarship Program
• (U.S. citizens only) Be added to an existing NSF grant awarded to a professor: NSF Research Experience for Undergraduates
• (U.S. citizens and permanent residents) Blue Waters student internship program (1-year program)

Graduate Students

• (U.S. citizens and permanent residents) Blue Waters Graduate Fellowship Program
• University of Utah Financial Aid and Scholarships
• U of U Graduate Fellowships
• The College of Engineering list of third-party scholarships
• Conference student paper contests
• (For U.S. citizens) NASA Space Technology Research Fellowship
• The U of U Alumni Association scholarship list
• (For U.S. citizens) National Science Foundation Graduate Research Fellowship
• IEEE Antennas and Propagation Society Pre-Graduate (Pre-Ph.D.) and Graduate Research Awards (due April 1 and Nov. 1)
• IEEE Antennas and Propagation Society Design Contest
• IEEE Microwave Theory and Techniques Society Pre-Graduate (Pre-Ph.D.) Scholarships
• IEEE Microwave Theory and Techniques Society Graduate (Ph.D.) Fellowship Awards
• (For U.S. citizens and permanent residents) Semiconductor Research Corporation
• IBM Ph.D. Fellowship Awards Program
• Microsoft Research Ph.D. Fellowship Program
• (For U.S. citizens) Truman Fellowship (3-year fellowship in national security science and engineering, for recent Ph.D. graduates)
• (For U.S. citizens) Department of Energy Graduate Fellowship Program
• Summa Foundation Graduate Fellowships in Advanced Electromagnetics
• (For U.S. citizens) Hertz Fellowship
• (For U.S. citizens) Science, Mathematics and Research for Transformation (SMART) Scholarship (Department of Defense)
• (For women) American Association of University Women Fellowships and Grants
• (For women) Microsoft's Graduate Women's Scholarship Program
• (For women) Fellowship, scholarship, and award opportunities listed with the Society of Women Engineers
• (For minorities) National GEM Consortium M.S. Engineering Fellowship
• (For minorities) NASA Harriett G. Jenkins Predoctoral Fellowship
• (For minorities and U.S. citizens) Be added to an existing NSF grant awarded to a professor: NSF Graduate Research Supplement (renewable for up to 3 years)

Post-Docs

• (For women) L'Oréal USA Fellowships for Women in Science

Recommended Courses (working list)

You should take classes that will best prepare you for your research topic and future career. Discuss these with Prof. Simpson, and make her aware of topics or areas you are not yet comfortable with, so that we can plan the best course of study for you. Below are some courses to consider; bold ones are especially recommended.

Undergraduates

• ECE 4900, Research Credit
• ECE 3300, Fundamentals of Electromagnetics and Transmission Lines
• ECE 5340, Numerical Methods in Electromagnetics
• CS 4230, Parallel Programming
• ECE 5320, Microwave Engineering I
• ECE 5321, Microwave Engineering II
• ECE 5324, Antenna Theory and Design
• ECE 5962, Metamaterials and Advanced Antenna Theory
• ECE 5411, Optical Communication Systems
• ECE 5480, Principles of Ultrasound
• ECE 5962, Optics for Energy
• ECE 5960, Fabrication and Characterization Techniques for Nanostructures
• GEO 5240, Physical Fields II: Electromagnetic Methods
• GEO 5250, Inversion Theory and Applications

Graduate Students

• ECE 6970/7970: If you are an RA (doing research on a funded project), register for 6970 or 7970 research credits. The maximum is 9 credits per semester of 6970/7970, but register for fewer at the beginning of your program and more later on, because the tuition benefit pays for courses for the first 84 credits. Talk to a department graduate advisor to make sure you're signing up for the correct credits.
• If English is a second language: consider taking EAS 6050, Advanced Spoken English, as needed
• ECE 6340, Numerical Methods in Electromagnetics
• ECE 6322, Microwave Engineering I
• ECE 6323, Microwave Engineering II
• ECE 6324, Antenna Theory and Design
• ECE 6310, Advanced Electromagnetic Fields
• MATH 5620, Numerical Analysis 2
• CS 6230, Parallel Programming HPC
• CS 6963, Parallel Programming for GPUs
• CS 6235, Parallel Programming for Many-Core Architectures
• CS 6160, Computational Geometry
• GEO 6240, Physical Fields II: Electromagnetic Methods
• GEO 6250, Inversion Theory and Applications
• For optics, biophotonics, nano:
  • ECE 6461, Nanophotonics
  • ECE 6460, Biophotonics
  • ECE 6451, Nonlinear Optics
  • ECE 6450, Ultrafast Optics
  • ECE 6440, Integrated Optics and Optical Sensors
  • ECE 6430, Statistical Optics, Interferometry, and Detection
  • ECE 6420, Fourier Optics and Holography
  • ECE 6961, Optics for Energy
  • ECE 6960, Fabrication and Characterization Techniques for Nanostructures

Ph.D. Qualifying Exam Hints

• Your presentation should take about 35 minutes (timed without questions from the committee). Expect many questions from the committee during your presentation.
• Keep in mind that there could be errors in the papers you are reviewing. Don't assume everything in the papers is correct.
• Do not just regurgitate what is in the papers. Instead, understand and explain how, why, the significance of the research, its relation to your research, etc.
• Do not necessarily show every equation and figure published in the papers. Determine the main points you want to describe, and show only what is needed to support them.
• Understand where the equations in the papers come from, and why the authors chose to use them. Also understand each variable and the form of the equations.
• Understand the figures you put in your presentation. Does the behavior of the waveforms make sense? Why and how did the authors get those results? How would the results change with different values or parameters?
• Read through, and study in detail as needed, the references of the papers.
• What follow-on / future work could be performed relating to the content of the papers?

Typical Conferences We Attend

• IEEE Antennas and Propagation International Symposium and USNC-URSI National Radio Science Meeting (rotates locations; see the list of upcoming conferences). Note the abstract / paper submission deadline is in mid-January!
• USNC-URSI National Radio Science Meeting, held annually in Boulder, CO in January (conference website). Note the abstract submission deadline is in September, including for the student paper competition. Don't forget to apply for student travel support when submitting your abstract.
• Other conferences attended by our group depending on the topic of your project, such as the AGU Fall Meeting each December or the International Conference on Electromagnetics in Advanced Applications (ICEAA).

Summer School Options

• NASA Heliophysics Summer School
• Incoherent Scatter Radar Summer School
• Los Alamos Space Weather Summer School

Presenting

• Ask yourself:
  • What is my main point? (What do I want audience members to remember most if they only remember one thing?)
  • What is the fundamental purpose of my talk? Why is my work important?
  • How much time do I have to present? (Typically 17 minutes at conferences, leaving 3 minutes for questions.)
  • Who is the audience? What is their background? How can I tailor my talk to this specific audience?
  • Why was I asked to speak? Or, why did I decide to speak?
  • What visual medium is most appropriate for this situation and audience? (Typically slides or posters at conferences.)
• It is often a good idea to include your primary result(s) toward the beginning of your talk, before diving into the details of how you obtained them.
• Know your material inside and out.
• Practice, practice, practice! Practice in front of someone, or at a minimum standing up in front of an empty room as if your audience is there. Make sure you can get through your material in the time allotted without rushing.
• Don't overwhelm your audience with equations. It will be difficult for them to absorb too many in one presentation.

Publishing

• If you used CHPC resources to obtain your results, include an acknowledgement: “The support and resources from the Center for High Performance Computing at the University of Utah are gratefully acknowledged.”
• If Prof. Simpson used research funds to support you while you performed the research, include an acknowledgement to the funding source.
• If your first language is not English, please meet with the Writing Center to get the English in your paper corrected (unless your English writing skills are very strong), and / or have a native English speaker review your document before giving it to Prof. Simpson to review. (Ask them to keep the manuscript confidential!)
• If you want positive reviews, think about what the reviewers need: easy to read and understand, no typos, and they need to believe and get excited about your work. Remember, they are volunteering their time to review your paper! Think about why they review — they get excited about good ideas and care about the field.
• It should go without saying that you must not copy any text, figures, etc. from any source without citing it. Also, do not use identical text from a source even if it is cited! Write what you want to say in your own words.
• Before giving Prof. Simpson a draft paper to review, check the document with a spell checker. As you progress through your studies, Prof. Simpson will likely need to make fewer corrections and changes to your papers.
• Don't be discouraged by negative reviews. Some reviewers are very critical, even of good work. An identical paper can get both fantastic and horrid reviews, depending on who reviews it and their background, expertise, and demeanor. View their comments as constructive feedback as much as possible, and revise and resubmit!

Proposal Writing

• Grantsmanship = salesmanship. Sell an idea to a funding agency based on a critical need within its mission.
• Personalize / tailor your biosketch for each proposal.
• Meet the reviewers' needs: easy to read, no typos, clearly point out the innovation and importance of the work, make them believe. Remember, they are volunteering their time to review your proposal! Think about why they review — they get excited about good ideas and care about the field.
• For the budget, propose what would reasonably be needed without going over the maximum amount (if there is one). If no maximum is stated, propose what you would reasonably need, and the funding agency will cut it down if needed.
• How will you measure success?
• Include letters of support for collaborations.
• For large projects, you could propose an advisory board for the project that meets every year or so.
• For NSF:
  • Include what science questions you're going to address.
  • Usually, three objectives are clearly written out.
• For NIH:
  • NIH comprises 27 different institutes and centers. Submit to the one that best fits the proposed work.
  • You need a detailed clinical applicability / feasibility plan.
  • Check the proposal assignment 2–3 weeks after submitting to make sure it was assigned to the correct institute / center.
  • Two strikes and you're out: if a proposal is declined twice, you can't submit another proposal on the same topic. If the first one is declined, it's hard to recover, because reviewers of the second proposal know the results of the first.
  • Only the top 40–50% are even discussed.
  • Specific Aims — make a promise: state the problem, significance, innovation, and how you are uniquely qualified; state the aims (going to learn ** by doing **); state the outcome / impact. Remember, most reviewers will only see your specific aims! Also, your aims should not depend on each other.
  • Research Strategy — deliver on that promise: Significance (½ to 1 page, tailored to the mission of each aim); Innovation (½ page: why original and exciting, how it will create a paradigm shift, creative, uniquely qualified to perform the work); Approach (10 pages, or 5 for shorter proposals: relevant preliminary data, experimental approach, strategy or approach, aim, hypothesis, experimental design, methods, preliminary data, potential problems, expected results, impact).

FDTD and Debugging Notes

• Compile all production codes with the flag -O3 (or equivalent) so the code runs faster.
• Debugging:
  • Save different versions of your code, even while developing it. Then you can go back to an earlier version if you suddenly get something like a segmentation fault you can't fix (i.e., you “break” your code, which sometimes happens).
  • Always use implicit none in your FORTRAN programs. This ensures your program doesn't create a new variable every time a variable name is misspelled.
  • Make small changes at a time and carefully test the code after each one. If you make too many changes at once and the code doesn't work right or goes unstable, it can be very hard to find the problem, and you may waste a lot of time (or have multiple bugs). You actually save a lot of time by making small changes and testing after each.
  • The best way to debug is to run as few time steps as possible (even just 1 to 3 time steps if you're testing for symmetry). Note: results may not be perfectly identical / symmetric in the last 1–2 decimal places, even where you'd expect them to be, because of machine precision. If you aren't just testing for symmetry, you may want to run 20 or 50 time steps to make sure the source is where it should be and the generated EM wave is propagating correctly.
  • Useful flags (these slow down your code! Only use them when debugging): -check bounds to see if you're accessing an array beyond its size; -traceback for more info on an error; -warn all for additional warnings. (See also FORTRAN Compiler Notes.)
• Order your do loops (in FORTRAN) as k, j, i (3rd, 2nd, then 1st dimension of field_value(i,j,k)) for optimal computational performance.
• Double precision (64-bit real) = 8 bytes per number, ~15–16 significant figures. Single precision (32-bit real) = 4 bytes per number, ~7 significant figures.
• Only use about 75% of the available memory; reserve the rest for running the machine.
  (# bytes on computer) × 0.75 / (# bytes per number) → # numbers; divide by (# stored values per cell) → maximum # grid cells per processor.
  If you start using virtual memory, your code will slow down substantially or crash.
  So far, the largest code our group has run was on Encanto using 384 processing cores and a grid size of 2400 × 2400 × 1740.
• Accuracy: 1st-order → error drops linearly; the Yee grid is 2nd-order accurate; the hexagonal grid is 4th-order accurate.
• Dynamically allocate arrays when possible, to save memory when arrays aren't needed:

  real, pointer :: temp(:,:)
  allocate(temp(1:100,1:100))
  deallocate(temp)

• DDT is a symbolic, parallel debugger that allows graphical debugging of MPI and OpenMP applications.

MATLAB Notes

• Use the unwrap command to unwrap a phase plot that jumps by 2π.

ParaView (Plotting) Notes

• ParaView download
• VTK file formats (PDF)
• ParaView tutorial (PDF)

Linux Notes

• man command — open the manual page describing a command
• Some basic commands:
  • ls — list files in the directory
  • cd directoryname — change to that directory
  • Type the first few letters of a filename and hit TAB to autocomplete (as long as only one file matches)
  • mkdir newdirectoryname — create a new directory
  • mv oldfilename newfilename — rename a file (careful: overwrites newfilename if it already exists!)
  • mv filename directory — move a file into a directory (careful: overwrites a file of the same name there!)
  • cp filename1 filename2 — copy filename1 to filename2 (careful: overwrites filename2 if it already exists!)
  • top — display top CPU processes; inside top, press k and enter a process ID to kill that process
  • kill -9 PID — kill a process from the command line
  • history — list previously entered commands
  • rm -r directoryname — remove a directory even if it has files in it
  • diff filename1 filename2 — compare two files, showing line numbers and text where they differ
  • ps — see which shell is running (csh, ksh, bash)
  • setenv variable value — set an environment variable (csh/tcsh; in bash use export variable=value)
  • reset — reset the terminal if output is garbled
  • du -sk directoryname — see how much space a directory uses (in kB)
  • du -k directoryname — same, listing all files
  • du -sk * | sort -n -r | more — list directory sizes starting with the largest (in kB)
• Text editors: gvim, vim, emacs, pico, nano
• Copy files between computers using Fugu, scp (terminal), or WinSCP (Windows).
  • From the computer that has the file (keeping the same filename):
    scp filename username@computername:directory/.
  • From the computer you want to copy the file to (keeping the same filename):
    scp username@computername:directory/filename .
  • You can put a new filename instead of using “.”
  • Pay attention to which direction you are copying. If you use the same filename on both machines, you could accidentally overwrite the wrong file.

FORTRAN Compiler Notes

• Always check that no arrays are accessed out of bounds and get information about any errors with these flags. Remove them for production runs, because they slow the code down significantly!

  Intel:  -check bounds -traceback
  GNU:    -fbounds-check -fbacktrace
  PGI:    -C

• Time your code by running it for 100 (or more) time steps and seeing how long just the time-stepping takes. Use this to estimate how long a longer run will take. Note that updates take longer when there are non-zero field components, so for larger grids, run the test longer to estimate the time for more steps (since far from the source, components will still be zero for a number of time steps).
• Free FORTRAN compilers for your laptop, before using the supercomputers:
  • Silverfrost FTN95 (personal edition, with the Plato editor) for Windows — simple to install and use, if you are familiar with Windows but not Linux and run time isn't important.
  • gfortran — a fast compiler (about 10× faster run times than Silverfrost for my codes), but you must be familiar with Linux. You can also install Cygwin (a Unix-like environment for Windows) to use gfortran on Windows.
• ifort programname — compile with the Intel compiler.
• Use the flag -r8 (or the like) to run a code in double precision without changing every real number to double precision.
  Note: for MPI codes, you can use -r8, but you also need to change all send/recv commands to use MPI_REAL8 instead of MPI_REAL!
• An example online tutorial for learning FORTRAN (or just search “FORTRAN tutorial”): fortrantutorial.com
• With the Intel compiler, unit numbers 100, 101, and 102 are permanently associated with standard input, standard output, and standard error. Do not use these unit numbers for opening / closing your own output files.

Lab Computer Notes

• Access remotely using ssh or similar. In a terminal, type ssh username@computername
• ulimit -s unlimited — allows you to run a larger code than otherwise (just be careful).
• To log out of the computer and keep your code running:

  nohup ./a.out > run.dat &

  (Anything that would be written to the screen goes into the file run.dat.)
• Running MPI codes on lab computers. Do this once only:

  cd $HOME
  touch .mpd.conf
  chmod 600 .mpd.conf
  vim .mpd.conf
     (in this file, type:   MPD_SECRETWORD=fdtd)

  After doing the above, or any other time you want to run an MPI code, type:

  mpd &

  Then compile and run your MPI code. Other useful commands:

  ps -aef | grep mpd                          # check that the mpd daemon is running
  cat /proc/cpuinfo | grep processor | wc -l  # how many cores are available
  mpirun -np 4 ./a.out                        # run on 4 processors
  mpiexec -np 4 ./a.out                       # ...or this, depending on the machine

Parallelization Notes

• If you need to use fewer processors than you reserved, use setenv NPROCS 128 or similar (e.g., if reserving 130 processors but using 128 in the model).
• DDT is a symbolic, parallel debugger that allows graphical debugging of MPI and OpenMP applications.
• To debug, you can also use the TotalView debugger. If using a Mac, log in with -X to turn on X11 tunneling.

MPI

• The CHPC on campus posts slides on its website and periodically runs info sessions on using MPI. Take advantage of these resources, and get on the CHPC listserv to learn about upcoming events.
• Compile using:

  mpif90 programname
  mpif90 datadefine.f subroutines.f mpiFDTD3d.f

• Time part of a run (make sure to declare t1 and t2!):

  t1 = MPI_WTIME()
     ...
  t2 = MPI_WTIME()
  write(*,*) "Total time (in seconds) =", t2-t1

• You can try MPI_BARRIER(MPI_COMM_WORLD,ierr) to synchronize all processes before continuing, but it is known to not always work correctly. Also, if not all processes reach this command (for example, if it's inside an if statement that isn't true for all processes), the code will hang.
• Make sure to have call MPI_FINALIZE(ierr) at the end of your code, or you won't leave the supercomputer in a good state and the IT folks will get upset.
• To monitor memory:

  qstat -m -u username
  ssh topnode
  top
  ctrl-c      # go back to home node
  exit

OpenMP (mostly for shared-memory computers)

• Compiler flags: pgf90 -mp; ifort -openmp; gcc -fopenmp
• For running codes on CPUs. You add simple parallelizing hints into your code as comments, written so that OpenMP sees them and uses them to parallelize your code for you. This avoids tedious coding with MPI. The same OpenMP code can run on a single processor (since the commands are comments) as well as on a computing cluster.
• Avoid nested “if” statements.

OpenACC

• For running codes on GPUs (known to run FDTD simulations faster than CPUs).
• Moving data between CPUs and GPUs is slow (a bottleneck) and should be kept to a minimum.
• Avoid nested “if” statements.

University of Utah Campus Supercomputer Notes

2014: If for some reason we don't have SUs, or you don't want to use our units, you can submit jobs to sanddunearch, a cluster that everyone with an account on Ember can use, and which doesn't use allocation units.

• Use as short a walltime as possible without risking your job not finishing in time (otherwise the scheduler will kill your job before it completes). Shorter walltimes help because the scheduler runs jobs out of order to backfill (fit in shorter jobs when longer jobs complete early).
• Always perform initial debugging and testing on your desktop before moving your code to the supercomputers.
• When logging on, use your UNID with a lowercase “u” in front.
• Store your results in a secure location when your job completes, and remove any files you no longer need. Some disk space is cleaned regularly, and you will lose your data and codes if you don't store your files in the correct directories.
• You need to request an account through the CHPC website to access any of these supercomputers. Prof. Simpson will be asked by CHPC to approve the request.
• In general, only read from or write data to files at the very beginning or end of your code. Avoid writing data every time step or during time-stepping (unless it's an isolated event), because this slows down your code considerably.
• Check out the CHPC website for more info on campus supercomputers and how to get access.
• To debug, you can use the TotalView debugger. If using a Mac, log in with -X to turn on X11 tunneling.

Math and Data Resources

Signal Processing

• Discrete-Time Signal Processing by Alan V. Oppenheim and Ronald W. Schafer (available from the library)
• Lecture series (YouTube)

Measurements and Data Access

• WALDO — Worldwide Archive of Low-Frequency Data and Observations
• The Madrigal Database — contains experiments across space physics
• YouTube videos about Madrigal
• AMISR incoherent scatter radar experiments
• Incoherent scatter radar resources (2024 ISR School)

GNU Plot

Good for plotting data on a node without having to open MATLAB, etc. See gnuplot.info.

plot sin(x)
plot [x1:x2] [y1:y2] <function>
set autoscale y
set logscale y
show logscale
set nologscale y
set title "File Graph"
save "mywork.gnu"
load "work.gnu"
plot "fileA.dt" using 1:2 title 'dataA'
set xlabel "label"
set xtics (0.02, 0.04, 0.06)
unset xtics
set xtics auto

# To print:
set out "file.ps"
load 'saveplot'
!lpr -Pteerlpl file.ps

Cloth Posters

Below are detailed steps to success with cloth posters. The procedure involves a lot of image processing, and it depends on which program you use to create the poster; some programs (MS Office) make it hard to get the poster files ready. These steps use spoonflower.com.

1. Create the poster in PowerPoint, Inkscape, or Adobe Illustrator just as you would for a paper printer. Take care when choosing the dimensions; not everything will work! Read the extra notes below to understand why.
2. Export the file as a PNG with at least 300 dpi. This will be a huge image file; for instance, a 42" × 42" poster gives an image of 12600 × 12600 pixels. There are many ways to export. There isn't a direct way in PowerPoint, but the most surefire way is:
   1. Export the poster as a PDF from whatever program you are using.
   2. Import the PDF into another graphics program, such as Photoshop or GIMP, at your desired resolution (300 pixels per inch recommended).
   3. Save the file as a PNG (this file will be large, e.g., 15–35 MB).
3. Now the tricky part: resize the image to fit the cloth properly. For example, with a W × H = 36" × 42" poster, where cloth is cut by the yard and is 56" wide (this depends on the material):
   1. Rotate the image so the width of the poster runs along the length of the cloth. The 36" dimension fits perfectly into a yard.
   2. Add white space to center the poster on the cloth as desired, making the file 56" × 36" (accounting for the rotation, which is why 36" appears in the height).
4. Go to spoonflower.com and click Create → Custom Fabric. Upload your file.
5. Under the Repeat selection, choose “Center.” Click “Design DPI” and type in the DPI used in your image.
6. Choose your fabric type, and make sure “Size and Amount” reads “Yards (56in width).” Enter the number of yards you want printed.
7. Click Add to Cart and finish the checkout.

Extra notes

Spoonflower only prints by whole yards, which limits the dimensions you can print. The width of the cloth depends on the material; performance knit, for example, is 56" wide. An original 42" × 42" poster would have required printing 2 yards, so it was reformatted to fit within W × H = 36" × 42", ensuring one dimension was at most a yard.

As in step 3, extra image processing was needed to make the other dimension equal 56", because Spoonflower does no extra cutting or cloth processing afterwards, and its online software is not flexible about configuring the image. The cloth arrives as 36" × 56", so there will be a lot of white space when the poster covers only 36" × 42". The extra processing forces the white space to the bottom of the poster, so it can simply be rolled up and pinned to the poster mounting board.