
   12.3. File Formats
   ==================

    12.3.1 Common Tools and Formats
    ===============================

      The most common internal file format is the output by MACHINE
      suboption. This is a semicolon seperated file format, which
      optionally supports a TITLE line for each field. The format is
      compatible to MS-Excel.
  
      The most important data collector and generator ctys-vdbgen is
      just a wrapper for the call of ctys with the action
      ENUMERATE. The replied data is transparantly stored in the main
      file database "enum.fdb", which could be inspected and edited by
      any ASC-II editor or a common spreadsheet application.
  
      The caching boosts the performance beginning with the factor of
      10, which could be much more, depending of various specific
      circumstances. Even though it could theoretically slow down, no
      practical example occured.
  
      The generated enum.fdb is just a raw static cache of distributed
      configuration data for VMs, which may lack some additional
      information. Therefore another file database is generated in
      order to map TCP/IP data of the Guest-OSs within VMs to their
      MAC-Addresses and DNS names. This file is called the macmap.fdb.
  
      It could be generated by the usage of the tools
      ctys-extractMAClst and/or ctys-extractARPlst.
  
      The prefetched databased "'.fdb" are pre-processed on a second
      level by aggregating and correlating the data into a common
      final static cache-database "ctysd-vhost.statcache.cdb". 
      Additional dynamic intermediate data my be cached within tmp
      directory.
  
      The databases could be listed and inspected by the tool
      ctys-vhost.

  
         - ctys-vhost -S list
           Lists source databases of first level cache "fdb", which
           includes the non-cached group files.
  
         - ctys-vhost -C list
           Lists statcache caches of second level prefetch, which
           includes the cached group files.
  

      Some additional options exist within ctys-vhost for listing of
      members and various output.


    12.3.2 Groups
    =============

     The groups concept as described within the introduction supports
     the assembly of arbitrate entities into a common entitiy, which will
     be handled as one logical instance. This includes the nesting of
     includes of groups.

     Therefore a file has to be supported in a defined subdirectory,
     with the name as the literal group name to be used within ctys
     arguments as a replacement for a <machine-address>.

     The group file supports the keyword "#include" which has to be
     left-most on a line followed by a groupname. The number of
     includes and the level of nesting is not limited. Recursion loops
     are checked and aborted by a pre-defined level.

     The subdirectory containing the group will be search from a given
     list of directories by the path variable "CTYS_GROUPS_PATH",
     which uses a colon seperated list of directory paths.

     The ordinary host entries could be one on each line, or a list of
     more than one comma-seperated entries on a line. For each entry
     could be context specific options set the same as on the
     commanline arguments.

     An exampel with nested incudes is supported in the install
     sub-directory conf/ctys/groups.

     Following is an example with context options.

     
       #
       #This groups contains all machines supporting VMs to be used by
       #user acue.
       #
       #Almost any machine is currently accessible by NFS, thus allows
       #a simple means of load-balancing throughout the memgers of
       #this group.
       #
       #The only distinction has to be done by type of VM, which
       #depends on the actual#running kernel.
       #
       #
       #REMINDER: Any option is dominant from it's first occurance on,
       #          until overwritten. This is one of the limits of
       #          missing namespaces and smart usable assembled data
       #          structures within bash. But anyway the real huge
       #          benfit from using bash is the opportunity for almost
       #          anyone to add his own modules by simple shell scripts.
       #
       #          It has to be recognized, that due to some basic
       #          requirements - e.g. grouping of potential
       #          Desktop/Workspace display, the jobs are re-ordered.

       #          So it is recommended, that when any context-option
       #          is required, that ALL might be assigned it's own
       #          options.
       #
       #
       #          So, it's accepted!
       #
       #
       #OpenBSD PM, for tests only. Supports no VMs, Just OpenBSD-PM and 
       #common HOSTs-(CLI,X11,VNC)
       root@host1'(-a enumerate=machine)'

       #Multi-Display-WS with local emulation for usage of
       #Win-Equipment, such as Dymo-LabelWriter 320 with it's own
       #drivers within VMWare and W2K.

       host2'(-a enumerate=machine)'

       #WMware WS wortkstation, DualOpteron-244
       host3'(-a enumerate=machine)',user5@host0'(-a enumerate=machine)'

       #Experimental Dual-PIII-Coppermine, yum-install of
       #VMware-server/player-variants

       host4'(-a enumerate=machine)',user3@host5'(-a enumerate=machine)'

       #Main fileserver with DualP-III-Thualatin
       host5'(-a enumerate=machine)',user3@host10'(-a enumerate=machine)'

       #Backup and Experimental HVM by Core2-Duo-6300
       host6'(-a enumerate=machine)',user2@host9'(-a enumerate=machine)'

       #Experimantal Celeron - S420
       root@host6'(-a enumerate=machine)'

       #Experimental Celeron 2,4GHz
       root@host7'(-a enumerate=machine)'


       #app1 as primary paravirtualized DualOpteron-244-Xen-Server
       host8'(-a enumerate=machine,b:/homen/userA%/home1/xen)'


     On the ctys call the group could be given context options, even
     when it's members have context options. But not the
     include-statements. Due to the philosophy, that the last wins,
     the outer . this is the CLI options - will replace options left
     of it, within the groups.


     When using ctys-vhost the groups will be cached as a pre-resolved
     subtree in common database (ENUMERATE) format. This will be
     recursively done for each group file as described in the
     following chapters.



    12.3.3. Static Import File Databases - fdb
     =========================================

    12.3.3.1. macmap.fdb
     ===================

      This file contains the output of the standard call to one of the
      tools "ctys-extractMAClst" or "ctys-extractARPlst". Which is a
      three colon semicolon seperated table, calle file-database. The
      record format is

        <DNS-name>;<MAC-address>;<TCP/IP-address>

      The default format is expected by the post-processing tools.


      The tools ctys-extractMAClst and ctys-extractARPlst could be
      used by themself for search and output to stdout
      too. 

      ctys-vhost allows by the flag "MACMAP" the optional usage of
      this database when only output suitable is selected for the "-o"
      option. Or allows the forced usage by "MACMAPONLY".

      ctys-macmap works natively on macmap.fdb.



    12.3.3.2. enum.fdb
     =================

      The enum.fdb file is the first level cached raw output from
      the ENUMERATE=MACHINE call. Therefore the call of ctys-vdbgen is
      used, which is a wrapper for the ctys call. Any option of ctys
      could be provided for ctys-vdbgen and will be passed through
      transparently. Therefore the flag "-T" and the given targets,
      e.g. as a group, could be used to constrain the collected and
      stored data. Using this features several databases could be used
      independently with different scopes of network view.


      The record format is as shown with the call
      "ENUMERATE=MACHINE,TITLE":

        "ContainingMachine;SessionType;Label;ID;UUID;MAC;TCP/IP;\
        VNCAccessPort;VNCBasePort;VNCDisplay;Distribution;OS;\
        VersionNr;SerialNr;Category"

        host.soho;QEMU;tst100;\
        /homen//tst100-01.01.001.x86_64/tst100-inst.conf;\
	;00:50:56:13:11:40;;;;;;;;;


      Space are not allowed, multiple entres will be ignored, just the
      first will be used.



    12.3.4. Static Pre-Fetch Cache Databases - cdb
    ==============================================

      The Pre-Fetch Caches are the second level, where some time
      consuming preprocessing and correlation is performed. This also
      includes the finale resolution of the group files by replacing
      each <machine-address> entry with the appropriate ENUMERATE
      result, and resolving the whole resulting include-tree.

      The common enum.fdb entries are stored within one database file,
      whereas the groups are resolved with their whole tree for each
      defined group file.

      The record format is the common ENUMERATE format.



    12.3.4.1. grpscache.cdb
    =======================

     The groups cache is basically the same as the enum.fdb, but the
     entries are for each group the reolved complete include tree.



    12.3.4.2. statcache.cdb
    =======================

     This is the cached enum.fdb. In addition to enum.fdb the
     macmap.fdb is correlated to any ITCP relevant field. Therefore -
     if available - any item should now contain a MAC-address and the
     related TCP/IP-address. This allows by simple search operations
     the evaluation of the GuestOSs TCP/IP-address from the VMs
     configuration file. Thus the native access to the GuestOS.



    12.3.5. Dynamic Runtime Cache Databases
    =======================================

     Some internal data is cached into files, which could be
     controlled by flags. This is due to reuse of data spanning
     more than one call.
 

    12.3.6. Configuration Entries
    =============================

     The various supported VMs have different configuration options
     for their machine files. They vary in the content and sysntax or,
     as for example QEMU, even do not support a configuration file,
     but an extended set of call options.

     Another common lack is that no entries related to the GuestOS is
     supported within the configuration files, but just some virtual
     hardware related like the MAC address and the UUID. This is
     perfectly allright, when just dealing with virtual hardware, but
     opens some challanges, when the running GuestOS has to be
     incorporated into a system, requiring native access.

     The same occurs, when a common management has to be defined for
     load distribution or relocation of services on different
     hardware.

     Therefore within the UnifiedSessionsManager a comon set of
     information required by the configuration of a VM and it's
     contained GuestOS is defined. This is just partly required to be
     defined when already supported by the VMs configuration file, but
     completely when lacking any information.

     Although the introduction of a new format may impyl some - but
     few - additional effort, the gain in combination with the tools
     like ctys-vhost and ctys-macmap are more than compensatory.

     The following keywords are defined and evalated by the ctys
     tools. The keywords are pre-fixed with "#@#" in order to provide
     the possibility to used them embedded into the specific
     confguration file when appropriate. Additionally some aother
     predefined files will be scanned.

     The basic convention for the naming and location of VMs
     configuration files is defined as:


       1. Any VM has it's own directory, wherein all related files are
          stored. These are the images and the configuration
          files. This is commonly pre-required for all VMs.

       2. The name of the directory is the same as the VM name, which
          could be the display name as well as a DomU name. For the
          directory name some variations could be set, e.g. due to
          versioning or backups. The tools like ctys-vhost search by
          default for the first match by alphabetical order only,
          which eliminates the usage of backup files.

       3. The configuration file will be named the same as the
          containing directory, but with the appropriate post-fix,
          which is e.g. 'vmx' or 'conf'.

       4. Additional files, to be optionally used for ctys information
          when required are:

            a. Same filenamepath as the VMs config-file, but with the
               post-fix 'ctys'.

            b. The same file stored in the parent directory.

            c. When common definitions to be used, these finally are
               searched within the CTYSCONF file.


           The scan for resolution will be performed until a match
           occurs, thus the first match wins.


     The order of search is:


       1. The configuration file of the VM
          The scanning subsystem first tries to match standard
          information, second the ctys specific entry will be scanned.
       2. The ctys-file
       3. The parent directory
       4. The CTYSCONF file


     The following keywords are defined:


      #@#MAGICID-<plugin>
        A magicID to for classification of the managing plugin for
        this type.

      #@#LABEL
        Label of VM, which is used as unique name. Could be e.g. the
        display name or a DomU name.

      #@#SESSIONTYPE
        The tye of session. This field will be evaluated by the plugin
        itself, passed through to the main dispatcher. It is
        frequently not required to support different values, when the
        plugin des not need to distinguish. 

        In case of emulators, namely QEMU, it could be helpful to
        distinguish between the emulated CPUs and visualize this to the
        user.

      #@#UUID
       The UUID of the machine.

      #@#MAC[0-9]*
       The MAC addresses, currently only MAC0 will be utilized.
  
      #@#IP[0-9]*
       The IP addresses, currently only IP0 will be utilized.

      #@#VNCACCESSPORT
       The port number of VNC access for the GuestOS, if required to
       be defined explicitly.

      #@#VNCBASEPORT
       The baseport when to be calculated with usage of display.

      #@#VNCACCESSDISPLAY
       The DISPLAY when required to be fixed.

      #@#DISTRO
       The distribution name of the GuestOS.

      #@#DISTROREL
       The release of the distribution of the GuestOS.

      #@#OS
       The name of the OS.

      #@#VERSION
       The release of the OS, which is the kernel release for Linux.

      #@#SERNR
       A free serial number for the user administration requirements.

      #@#CATEGORY
       The category of this Machine. Currently VM or PM.



     Additional data could be provided for and by the specific
     plugin. This is required for example in the specific
     configuration file defined for the QEMU plugin by ctys. Which has
     to be wrapped due to it's call-option only interface, to be
     unified in accordance to the common usage with the remaning VMs.
