MAOS
Multithreaded Adaptive Optics Simulator
Loading...
Searching...
No Matches
Architecture

This page describes how the code is organized and how a maos simulation runs. See Fundamentals for the data types and Debugging for debugging.

Library Layering

The source is organized as layered libraries. Each layer depends only on the layers above it (this is also the SUBDIRS build order in Makefile.am):

  1. sys/: operating system layer. Memory accounting (mem.c), threads and the thread pool (thread.c, thread_pool.c), sockets (sock.c, sockio.c), file IO with the bin and fits formats and gzip/zstd compression (bin.c, zstdio.c), the scheduler client (scheduler_client.c), and per OS process information (process_*.c).
  2. math/: numerical containers (dmat, cmat, smat, zmat, lmat, sparse dsp/csp, cell arrays, loc_t, map_t) with BLAS, LAPACK, FFTW, and CHOLMOD wrappers. The functions are type generic (see Type Generic Implementation).
  3. lib/: physics and AO building blocks reusable outside of maos, built into libaos together with sys and math: turbulence generation, ray tracing (accphi.c), the mkg, mkh, mkz, mkw operators, PCG, servo, PSD, pyramid WFS, the configuration file parser (readcfg.c), and the plotting client (draw.c). Functions here take basic types, not maos structs. lib/aolib.c builds bin/aolib.so, which is used by the Python (scripts/libaos.py) and Matlab interfaces.
  4. cuda/: GPU versions of the simulation and reconstruction, organized as cuda/math, cuda/recon, and cuda/sim. They mirror the CPU code paths and are compiled only when CUDA is found (USE_CUDA).
  5. maos/: the simulator.
  6. skyc/: sky coverage post processing, a separate executable with its own parameters (see Sky Coverage).
  7. tools/: the job scheduler, the monitor (GTK and browser interfaces), drawdaemon (GTK plotting), drawres, bin2fits, and other utilities.
  8. mex/ and scripts/: Matlab and Python bindings and scripts. scripts/lib2mex.py and scripts/lib2py.py parse the headers to generate mex/aolib.c and scripts/libaos.py, so the formatting of structs and doc comments in those headers matters.

Simulation Flow

The following is the call sequence of maos. Function names link to their documentation.

  • main() (maos/maos.c) parses the command line, then
    • setup_parms() (maos/parms.c) reads the configuration files into parms_t and checks for errors:
      • The readcfg_*() functions read every key, for example readcfg_sim(), readcfg_powfs(), readcfg_wfs(), readcfg_dm(), readcfg_tomo(), readcfg_gpu().
      • All keys that are read are saved to maos_HOST_PID.conf (linked as maos_recent.conf) in the output folder so that the simulation can be reproduced.
      • The setup_parms_postproc_*() functions derive dependent values and validate the parameters.
    • scheduler_start() waits for permission from the scheduler to start (see Scheduler, Monitor, and Drawdaemon).
    • setup_parms_gpu() selects the GPUs and sets the parms->gpu flags.
    • THREAD_POOL_INIT() initializes the thread pool or OpenMP with NTHREAD threads (option -n).
    • maos_setup() (maos/setup.c) sets up the AO system geometry and data structs. Its main steps are:
      • setup_aper(): the aperture grid and amplitude map (aper_t).
      • setup_powfs_init(): the WFS subaperture geometry (powfs_t).
      • setup_recon_prep(): the reconstruction grids (ploc, aloc, floc, xloc) and the ray tracing operators (recon_t).
      • setup_surf(): surface errors and NCPA.
      • setup_shwfs_phy(): physical optics WFS data (dtf, etf, i0, matched filter, cog).
      • setup_powfs_neasim(): gradient noise model for geometric WFS.
      • setup_powfs_calib(): NCPA calibration.
      • setup_recon_GA(), setup_recon_GF(), twfs_setup_GR(): actuator, focus, and radial mode to gradient operators.
      • ngsmod_prep(): low order NGS mode geometry for split tomography.
      • setup_recon_saneai(): noise covariance.
      • setup_recon_dither_dm(): DM dithering.
      • setup_recon_control(): tomography, DM fitting, or least squares reconstructor.
      • setup_recon_moao(): MOAO, if enabled.
      • setup_powfs_fit(): fitting of turbulence onto the WFS subapertures (sim.wfsalias=2 or sim.idealwfs=2).
      • setup_recon_mvm() or gpu_setup_recon_mvm(): the control matrix when recon.mvm is enabled.
      • gpu_wfsgrad_init(), gpu_setup_recon(), gpu_perfevl_init(), gpu_wfssurf2gpu(): copy data to the GPU when the corresponding parms->gpu flag is set.
      • setup_recon_misc(): PSD and other miscellaneous data.
      • plot_setup(): plot the geometry when plot.setup is set.
    • maos_sim() (maos/sim.c) runs the simulation for each seed in sim.seeds:
      • maos_iseed() calls sim_init() to create the runtime data (sim_t) and generates the frozen flow turbulence with genatm().
      • In the parallel (PARALLEL=1) and serial (PARALLEL=0) modes, maos_isim() is called for each time step from sim.start to sim.end. It calls prepare_isim() (turbulence for non frozen flow, sodium profile update, DM projection), then
        • with PARALLEL=1, queues reconstruct(), perfevl(), and wfsgrad() to run concurrently, waits for them, then calls shift_grad() and filter_dm(). The GPU versions of perfevl and wfsgrad are queued first so that they start early.
        • with PARALLEL=0 in closed loop, calls perfevl(), wfsgrad(), reconstruct(), shift_grad(), and filter_dm() in sequence.
        • with PARALLEL=0 in open loop, calls wfsgrad(), shift_grad(), reconstruct(), filter_dm(), and perfevl() in sequence.
        • postproc_isim() updates the reconstructor when requested (for example by Cn2 estimation or dithering).
        • print_progress() reports the progress.
      • In the event driven mode (PARALLEL=2), three loops run concurrently for all time steps and synchronize with each other through counters:
        • perfevl() and print_progress(),
        • wfsgrad() and shift_grad(),
        • reconstruct(), filter_dm(), and postproc_isim(), which updates the reconstructor when requested (for example by Cn2 estimation).
      • sim_free() frees the runtime data of the seed.
    • maos_reset() and free_parms() free the remaining data.

The main functions of each time step are:

  • perfevl() (maos/perfevl.c): ray traces the turbulence and DM shapes to the science directions and computes the wavefront error and PSFs.
  • wfsgrad() (maos/wfsgrad.c): computes the WFS OPD, the subaperture images in physical optics mode, and the gradients.
  • shift_grad(): copies the new gradients to gradlastcl/gradlastol for the reconstructor.
  • reconstruct() (maos/recon.c): computes the DM error signal from the gradients.
  • filter_dm() (maos/filter.c): applies the servo filter to compute the DM commands (dmcmd) and the actual DM shape (dmreal), including hysteresis (see DM Hysteresis).

In closed loop, reconstruct() uses the gradients of the previous time step (simu->reconisim is isim-1), which is what allows it to run concurrently with wfsgrad() and perfevl().

PARALLEL is read from the environment variable MAOS_PARALLEL (default 1; see read_sim_env()). maos_setup() changes it to 0 in open loop (sim.closeloop=0) or when evl.tomo is set. PARALLEL=2 requires OpenMP and does not support sim.dmproj; otherwise it falls back to 1. The environment variables MAOS_NO_WFS, MAOS_NO_EVL, and MAOS_NO_RECON skip the corresponding calls, which is useful for timing.

Key Data Structures

  • parms_t (maos/parms.h): all the input parameters, read once by setup_parms() and treated as constant afterwards. The members mirror the configuration keys, for example parms->powfs[ipowfs] (powfs_cfg_t), parms->wfs[iwfs] (wfs_cfg_t), parms->dm[idm] (dm_cfg_t), parms->sim (sim_cfg_t), and parms->gpu (gpu_cfg_t).
  • aper_t (maos/types.h): the telescope aperture: the evaluation grid (locs), the amplitude, and the PSF computation data (locfft).
  • powfs_t: data shared by all WFS of the same type (geometry, amplitude, and physical optics data such as the subaperture OTFs and matched filters).
  • recon_t: the reconstruction grids and operators (ploc, aloc, xloc, GA, GX, HXF, ...), the tomography and fitting parameters, and the control matrix.
  • sim_t: the runtime data of one seed: atmosphere, DM commands and shapes, gradients, performance results, saved telemetry, and the synchronization state for the parallel modes.
  • global_t: holds pointers to the above (global->parms, global->powfs, global->aper, global->recon, global->simu) for use by the scheduler listener, the signal handler, and the Matlab/Python interfaces.

GPU Code Paths

When maos is compiled with CUDA, the cuda/ library provides GPU versions of the WFS (gpu_wfsgrad*), performance evaluation (gpu_perfevl*), and reconstruction (gpu_setup_recon, gpu_tomo, gpu_fit, ...). Each task uses the GPU when its flag in parms->gpu is set: gpu.wfs, gpu.evl, gpu.tomo, gpu.fit, gpu.lsr, gpu.psf, and gpu.moao (defaults in config/maos/sim.conf). setup_parms_gpu() turns off the flags that do not apply to the configuration, sets gpu.recon, initializes the GPUs with gpu_init(), and clears all the flags if no GPU is used. The command line option -g selects the GPUs (-g-1 disables GPU) and -G sets the number of GPUs. The CPU code calls the gpu_* functions inside #if USE_CUDA blocks depending on these flags.

Adding a Parameter

Every runtime option is a key=value entry in the config/maos/*.conf files and maps 1:1 to a field of parms_t: wfs.thetax becomes parms->wfs.thetax (or parms->wfs[iwfs].thetax for per WFS keys). To add a parameter:

  1. Add the field with a doc comment to the struct in maos/parms.h.
  2. Read it in the matching readcfg_*() function in maos/parms.c with one of the READ_* macros, for example READ_INT(sim.foo), READ_DBL(sim.foo), READ_STR(sim.foo), or READ_DMAT(sim.foo), or READ_POWFS(dbl,foo), READ_WFS(dbl,foo), READ_DM(dbl,foo) for array parameters that have one value per powfs, WFS, or DM.
  3. Derive dependent values and validate the input in the appropriate setup_parms_postproc_*() function.
  4. Add the key with its default value and a comment to the appropriate .conf file, for example sim.conf, recon.conf, dbg.conf, or powfs_common.conf.
  5. Free any allocated memory in free_parms().

The configuration parser (lib/readcfg.c) checks the keys when it is closed in setup_parms(): a key that is supplied by the user (on the command line or in a user .conf file) but never read is an error, and so is a key that is read more than once. Unused keys in the default configuration files only produce a warning. Set MAOS_PERMISSIVE=1 to turn these errors into warnings. A key that is read but not found produces a warning and is assumed to be 0, so always provide a default in the .conf files. Conf files include= each other; later values override earlier ones, and key+=value appends to an array.

Threading

maos parallelizes both across the main tasks of a time step (see above) and inside each task. The macros in sys/thread.h provide a common interface:

  • QUEUE(group, fun, arg, njob, urgent) queues njob calls of fun(arg); WAIT(group, urgent) waits for the queued jobs in the group; CALL() queues and waits.
  • QUEUE_THREAD(), CALL_THREAD() run an array of thread_t, which divides an index range among the threads (see thread_prep()).
  • OMP_FOR(), OMP_TASK_FOR() parallelize loops with OpenMP.

When compiled with OpenMP 4.5 or newer, these macros use OpenMP tasks. Otherwise they use the thread pool in sys/thread_pool.c, which keeps a fixed number of threads that take jobs from a queue; a thread that waits for its jobs runs other pending jobs in the meantime. main() runs maos_sim() inside an OpenMP parallel/single region (OMPTASK_SINGLE) so that the tasks can be scheduled.

Memory Accounting

sys/mem.h replaces malloc, calloc, realloc, and free (in C code) with wrappers that record every allocation when memory debugging is enabled. It is enabled by default in debug builds (--enable-debug) and can be set with the environment variable MAOS_MEM_DEBUG=1 or 0. At exit, the unreleased memory is printed together with the backtrace of its allocation. MAOS_MEM_VERBOSE=1 prints every allocation and free. Use mycalloc(n, type) and mymalloc(n, type) to allocate typed memory. See Debugging.

Logging

The logging macros are defined in sys/common.h and filtered by the environment variable MAOS_LOG_LEVEL (default 0, range -5 to 5). The output goes to the terminal (unless detached) and to the log file.

  • info3(), info2(), info(): information of decreasing importance, shown at the default level.
  • dbg(), dbg2(), dbg3(): debugging information, shown only when MAOS_LOG_LEVEL is at least 1, 2, or 3, respectively.
  • warning(): a warning with the file and line number.
  • error(): prints the message with the file and line number and raises SIGUSR2. The signal handler (sys/mem.c) prints a backtrace and calls the handler registered by the program, which normally terminates it.
  • Variants such as info_once(), warning_once(), dbg_time(), and info_time() print only once or add a time stamp.

Scheduler, Monitor, and Drawdaemon

  • The scheduler (tools/scheduler.c) is a per user daemon that queues jobs and keeps their status. It listens on a TCP port that is derived from the user name or read from ~/.aos/port. When compiled with libwebsockets, it also serves the browser monitor (monitor.html) on that port plus 100. maos launches the scheduler if it is not running.
  • maos talks to the scheduler through sys/scheduler_client.c: scheduler_report_path() reports the job, scheduler_start() waits until enough CPUs are available (the -f option skips the wait), scheduler_report() sends the status every step, and scheduler_finish() reports the end. scheduler_listen() starts a thread in maos that handles requests forwarded by the scheduler, such as a request to start drawing.
  • The monitor (tools/monitor*.c, or monitor.html in a browser) connects to the schedulers on the hosts listed in ~/.aos/hosts, shows the jobs, and can kill jobs or request plots.
  • Plotting uses lib/draw.c. When plotting is requested (plot.* options or from the monitor), maos sends the data over a socket to drawdaemon (tools/drawdaemon*.c), which is either launched locally or connected through the monitor and the scheduler.

Configure with --disable-scheduler (implied by --enable-release) to build without the scheduler.