![]() |
MAOS
Multithreaded Adaptive Optics Simulator
|
This page describes how the code is organized and how a maos simulation runs. See Fundamentals for the data types and Debugging for debugging.
The source is organized as layered libraries. Each layer depends only on the layers above it (this is also the SUBDIRS build order in Makefile.am):
sys/: operating system layer. Memory accounting (mem.c), threads and the thread pool (thread.c, thread_pool.c), sockets (sock.c, sockio.c), file IO with the bin and fits formats and gzip/zstd compression (bin.c, zstdio.c), the scheduler client (scheduler_client.c), and per OS process information (process_*.c).math/: numerical containers (dmat, cmat, smat, zmat, lmat, sparse dsp/csp, cell arrays, loc_t, map_t) with BLAS, LAPACK, FFTW, and CHOLMOD wrappers. The functions are type generic (see Type Generic Implementation).lib/: physics and AO building blocks reusable outside of maos, built into libaos together with sys and math: turbulence generation, ray tracing (accphi.c), the mkg, mkh, mkz, mkw operators, PCG, servo, PSD, pyramid WFS, the configuration file parser (readcfg.c), and the plotting client (draw.c). Functions here take basic types, not maos structs. lib/aolib.c builds bin/aolib.so, which is used by the Python (scripts/libaos.py) and Matlab interfaces.cuda/: GPU versions of the simulation and reconstruction, organized as cuda/math, cuda/recon, and cuda/sim. They mirror the CPU code paths and are compiled only when CUDA is found (USE_CUDA).maos/: the simulator.skyc/: sky coverage post processing, a separate executable with its own parameters (see Sky Coverage).tools/: the job scheduler, the monitor (GTK and browser interfaces), drawdaemon (GTK plotting), drawres, bin2fits, and other utilities.mex/ and scripts/: Matlab and Python bindings and scripts. scripts/lib2mex.py and scripts/lib2py.py parse the headers to generate mex/aolib.c and scripts/libaos.py, so the formatting of structs and doc comments in those headers matters.The following is the call sequence of maos. Function names link to their documentation.
maos/maos.c) parses the command line, thenmaos/parms.c) reads the configuration files into parms_t and checks for errors:readcfg_*() functions read every key, for example readcfg_sim(), readcfg_powfs(), readcfg_wfs(), readcfg_dm(), readcfg_tomo(), readcfg_gpu().maos_HOST_PID.conf (linked as maos_recent.conf) in the output folder so that the simulation can be reproduced.setup_parms_postproc_*() functions derive dependent values and validate the parameters.parms->gpu flags.THREAD_POOL_INIT() initializes the thread pool or OpenMP with NTHREAD threads (option -n).maos/setup.c) sets up the AO system geometry and data structs. Its main steps are:sim.wfsalias=2 or sim.idealwfs=2).recon.mvm is enabled.parms->gpu flag is set.plot.setup is set.maos/sim.c) runs the simulation for each seed in sim.seeds: PARALLEL=1) and serial (PARALLEL=0) modes, maos_isim() is called for each time step from sim.start to sim.end. It calls prepare_isim() (turbulence for non frozen flow, sodium profile update, DM projection), thenPARALLEL=1, queues reconstruct(), perfevl(), and wfsgrad() to run concurrently, waits for them, then calls shift_grad() and filter_dm(). The GPU versions of perfevl and wfsgrad are queued first so that they start early.PARALLEL=0 in closed loop, calls perfevl(), wfsgrad(), reconstruct(), shift_grad(), and filter_dm() in sequence.PARALLEL=0 in open loop, calls wfsgrad(), shift_grad(), reconstruct(), filter_dm(), and perfevl() in sequence.postproc_isim() updates the reconstructor when requested (for example by Cn2 estimation or dithering).PARALLEL=2), three loops run concurrently for all time steps and synchronize with each other through counters:
The main functions of each time step are:
maos/perfevl.c): ray traces the turbulence and DM shapes to the science directions and computes the wavefront error and PSFs.maos/wfsgrad.c): computes the WFS OPD, the subaperture images in physical optics mode, and the gradients.gradlastcl/gradlastol for the reconstructor.maos/recon.c): computes the DM error signal from the gradients.maos/filter.c): applies the servo filter to compute the DM commands (dmcmd) and the actual DM shape (dmreal), including hysteresis (see DM Hysteresis).In closed loop, reconstruct() uses the gradients of the previous time step (simu->reconisim is isim-1), which is what allows it to run concurrently with wfsgrad() and perfevl().
PARALLEL is read from the environment variable MAOS_PARALLEL (default 1; see read_sim_env()). maos_setup() changes it to 0 in open loop (sim.closeloop=0) or when evl.tomo is set. PARALLEL=2 requires OpenMP and does not support sim.dmproj; otherwise it falls back to 1. The environment variables MAOS_NO_WFS, MAOS_NO_EVL, and MAOS_NO_RECON skip the corresponding calls, which is useful for timing.
maos/parms.h): all the input parameters, read once by setup_parms() and treated as constant afterwards. The members mirror the configuration keys, for example parms->powfs[ipowfs] (powfs_cfg_t), parms->wfs[iwfs] (wfs_cfg_t), parms->dm[idm] (dm_cfg_t), parms->sim (sim_cfg_t), and parms->gpu (gpu_cfg_t).maos/types.h): the telescope aperture: the evaluation grid (locs), the amplitude, and the PSF computation data (locfft).ploc, aloc, xloc, GA, GX, HXF, ...), the tomography and fitting parameters, and the control matrix.global->parms, global->powfs, global->aper, global->recon, global->simu) for use by the scheduler listener, the signal handler, and the Matlab/Python interfaces.When maos is compiled with CUDA, the cuda/ library provides GPU versions of the WFS (gpu_wfsgrad*), performance evaluation (gpu_perfevl*), and reconstruction (gpu_setup_recon, gpu_tomo, gpu_fit, ...). Each task uses the GPU when its flag in parms->gpu is set: gpu.wfs, gpu.evl, gpu.tomo, gpu.fit, gpu.lsr, gpu.psf, and gpu.moao (defaults in config/maos/sim.conf). setup_parms_gpu() turns off the flags that do not apply to the configuration, sets gpu.recon, initializes the GPUs with gpu_init(), and clears all the flags if no GPU is used. The command line option -g selects the GPUs (-g-1 disables GPU) and -G sets the number of GPUs. The CPU code calls the gpu_* functions inside #if USE_CUDA blocks depending on these flags.
Every runtime option is a key=value entry in the config/maos/*.conf files and maps 1:1 to a field of parms_t: wfs.thetax becomes parms->wfs.thetax (or parms->wfs[iwfs].thetax for per WFS keys). To add a parameter:
maos/parms.h.readcfg_*() function in maos/parms.c with one of the READ_* macros, for example READ_INT(sim.foo), READ_DBL(sim.foo), READ_STR(sim.foo), or READ_DMAT(sim.foo), or READ_POWFS(dbl,foo), READ_WFS(dbl,foo), READ_DM(dbl,foo) for array parameters that have one value per powfs, WFS, or DM.setup_parms_postproc_*() function..conf file, for example sim.conf, recon.conf, dbg.conf, or powfs_common.conf.The configuration parser (lib/readcfg.c) checks the keys when it is closed in setup_parms(): a key that is supplied by the user (on the command line or in a user .conf file) but never read is an error, and so is a key that is read more than once. Unused keys in the default configuration files only produce a warning. Set MAOS_PERMISSIVE=1 to turn these errors into warnings. A key that is read but not found produces a warning and is assumed to be 0, so always provide a default in the .conf files. Conf files include= each other; later values override earlier ones, and key+=value appends to an array.
maos parallelizes both across the main tasks of a time step (see above) and inside each task. The macros in sys/thread.h provide a common interface:
QUEUE(group, fun, arg, njob, urgent) queues njob calls of fun(arg); WAIT(group, urgent) waits for the queued jobs in the group; CALL() queues and waits.QUEUE_THREAD(), CALL_THREAD() run an array of thread_t, which divides an index range among the threads (see thread_prep()).OMP_FOR(), OMP_TASK_FOR() parallelize loops with OpenMP.When compiled with OpenMP 4.5 or newer, these macros use OpenMP tasks. Otherwise they use the thread pool in sys/thread_pool.c, which keeps a fixed number of threads that take jobs from a queue; a thread that waits for its jobs runs other pending jobs in the meantime. main() runs maos_sim() inside an OpenMP parallel/ (single regionOMPTASK_SINGLE) so that the tasks can be scheduled.
sys/mem.h replaces malloc, calloc, realloc, and free (in C code) with wrappers that record every allocation when memory debugging is enabled. It is enabled by default in debug builds (--enable-debug) and can be set with the environment variable MAOS_MEM_DEBUG=1 or 0. At exit, the unreleased memory is printed together with the backtrace of its allocation. MAOS_MEM_VERBOSE=1 prints every allocation and free. Use mycalloc(n,
type) and mymalloc(n, type) to allocate typed memory. See Debugging.
The logging macros are defined in sys/common.h and filtered by the environment variable MAOS_LOG_LEVEL (default 0, range -5 to 5). The output goes to the terminal (unless detached) and to the log file.
info3(), info2(), info(): information of decreasing importance, shown at the default level.dbg(), dbg2(), dbg3(): debugging information, shown only when MAOS_LOG_LEVEL is at least 1, 2, or 3, respectively.warning(): a warning with the file and line number.error(): prints the message with the file and line number and raises SIGUSR2. The signal handler (sys/mem.c) prints a backtrace and calls the handler registered by the program, which normally terminates it.info_once(), warning_once(), dbg_time(), and info_time() print only once or add a time stamp.scheduler (tools/scheduler.c) is a per user daemon that queues jobs and keeps their status. It listens on a TCP port that is derived from the user name or read from ~/.aos/port. When compiled with libwebsockets, it also serves the browser monitor (monitor.html) on that port plus 100. maos launches the scheduler if it is not running.maos talks to the scheduler through sys/scheduler_client.c: scheduler_report_path() reports the job, scheduler_start() waits until enough CPUs are available (the -f option skips the wait), scheduler_report() sends the status every step, and scheduler_finish() reports the end. scheduler_listen() starts a thread in maos that handles requests forwarded by the scheduler, such as a request to start drawing.monitor (tools/monitor*.c, or monitor.html in a browser) connects to the schedulers on the hosts listed in ~/.aos/hosts, shows the jobs, and can kill jobs or request plots.lib/draw.c. When plotting is requested (plot.* options or from the monitor), maos sends the data over a socket to drawdaemon (tools/drawdaemon*.c), which is either launched locally or connected through the monitor and the scheduler.Configure with --disable-scheduler (implied by --enable-release) to build without the scheduler.