PairedTTester (weka-stable 3.6.8 API)

java.lang.Object
- weka.experiment.PairedTTester

All Implemented Interfaces:: Serializable, OptionHandler, RevisionHandler, Tester

Direct Known Subclasses:: PairedCorrectedTTester

public class PairedTTester
extends Object
implements OptionHandler, Tester, RevisionHandler

Calculates T-Test statistics on data stored in a set of instances.

Valid options are:

 -D <index,index2-index4,...>
  Specify list of columns that specify a unique
  dataset.
  First and last are valid indexes. (default none)

 -R <index>
  Set the index of the column containing the run number

 -F <index>
  Set the index of the column containing the fold number

 -G <index1,index2-index4,...>
  Specify list of columns that specify a unique
  'result generator' (eg: classifier name and options).
  First and last are valid indexes. (default none)

 -S <significance level>
  Set the significance level for comparisons (default 0.05)

 -V
  Show standard deviations

 -L
  Produce table comparisons in Latex table format

 -csv
  Produce table comparisons in CSV table format

 -html
  Produce table comparisons in HTML table format

 -significance
  Produce table comparisons with only the significance values

 -gnuplot
  Produce table comparisons output suitable for GNUPlot

Version:: $Revision: 6431 $
Author:: Len Trigg (trigg@cs.waikato.ac.nz)
See Also:: Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`protected class`	`PairedTTester.Dataset` Utility class to store the instances pertaining to a dataset
`protected class`	`PairedTTester.DatasetSpecifiers` A list of unique "dataset" specifiers that have been observed
`protected class`	`PairedTTester.Resultset` Utility class to store the instances in a resultset

Field Summary

Fields
Modifier and Type	Field and Description
`protected int[]`	`m_ColOrder` The sorting of the columns (test base is always first)
`protected int[]`	`m_DatasetKeyColumns` An array containing the indexes of just the selected columns
`protected Range`	`m_DatasetKeyColumnsRange` The range of columns that specify a unique "dataset" (eg: scheme plus configuration)
`protected PairedTTester.DatasetSpecifiers`	`m_DatasetSpecifiers` The list of dataset specifiers
`protected int[]`	`m_DisplayedResultsets` An array containing the indexes of the datasets to display
`protected int`	`m_FoldColumn` The option setting for the fold number column (-1 means none)
`protected Instances`	`m_Instances` The set of instances we will analyse
`protected ResultMatrix`	`m_ResultMatrix` the instance of the class to produce the output.
`protected int[]`	`m_ResultsetKeyColumns` An array containing the indexes of just the selected columns
`protected Range`	`m_ResultsetKeyColumnsRange` The range of columns that specify a unique result set (eg: scheme plus configuration)
`protected FastVector`	`m_Resultsets` Stores a vector for each resultset holding all instances in each set
`protected boolean`	`m_ResultsetsValid` Indicates whether the instances have been partitioned
`protected int`	`m_RunColumn` The index of the column containing the run number
`protected int`	`m_RunColumnSet` The option setting for the run number column (-1 means last)
`protected boolean`	`m_ShowStdDevs` Indicates whether standard deviations should be displayed
`protected double`	`m_SignificanceLevel` The significance level for comparisons
`protected int`	`m_SortColumn` The column to sort on (-1 means default sorting)
`protected int[]`	`m_SortOrder` The sorting of the datasets (according to the sort column)

Constructor Summary

Constructors
Constructor and Description

PairedTTester()

Constructors
Constructor and Description
`PairedTTester()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`assign(Tester tester)` retrieves all the settings from the given Tester
`PairedStats`	`calculateStatistics(Instance datasetSpecifier, int resultset1Index, int resultset2Index, int comparisonColumn)` Computes a paired t-test comparison for a specified dataset between two resultsets.
`boolean`	`displayResultset(int index)` Checks whether the resultset with the given index shall be displayed.
`Range`	`getDatasetKeyColumns()` Get the value of DatasetKeyColumns.
`int[]`	`getDisplayedResultsets()` Gets the indices of the the datasets that are displayed (if `null` then all are displayed).
`String`	`getDisplayName()` returns the name of the tester
`int`	`getFoldColumn()` Get the value of FoldColumn.
`Instances`	`getInstances()` Get the value of Instances.
`int`	`getNumDatasets()` Gets the number of datasets in the resultsets
`int`	`getNumResultsets()` Gets the number of resultsets in the data.
`String[]`	`getOptions()` Gets current settings of the PairedTTester.
`ResultMatrix`	`getResultMatrix()` Gets the instance that produces the output.
`Range`	`getResultsetKeyColumns()` Get the value of ResultsetKeyColumns.
`String`	`getResultsetName(int index)` Gets a string descriptive of the specified resultset.
`String`	`getRevision()` Returns the revision string.
`int`	`getRunColumn()` Get the value of RunColumn.
`boolean`	`getShowStdDevs()` Returns true if standard deviations have been requested.
`double`	`getSignificanceLevel()` Get the value of SignificanceLevel.
`int`	`getSortColumn()` Returns the column to sort on, -1 means the default sorting.
`String`	`getSortColumnName()` Returns the name of the column to sort on.
`String`	`getToolTipText()` returns a string that is displayed as tooltip on the "perform test" button in the experimenter
`String`	`header(int comparisonColumn)` Creates a "header" string describing the current resultsets.
`protected void`	`initResultMatrix()` clears the content and fills the column and row names according to the given sorting
`Enumeration`	`listOptions()` Lists options understood by this object.
`static void`	`main(String[] args)` Test the class from the command line.
`String`	`multiResultsetFull(int baseResultset, int comparisonColumn)` Creates a comparison table where a base resultset is compared to the other resultsets.
`String`	`multiResultsetRanking(int comparisonColumn)` returns a ranking of the resultsets
`String`	`multiResultsetSummary(int comparisonColumn)` Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
`int[][]`	`multiResultsetWins(int comparisonColumn, int[][] nonSigWin)` Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
`protected void`	`prepareData()` Separates the instances into resultsets and by dataset/run.
`String`	`resultsetKey()` Creates a key that maps resultset numbers to their descriptions.
`void`	`setDatasetKeyColumns(Range newDatasetKeyColumns)` Set the value of DatasetKeyColumns.
`void`	`setDisplayedResultsets(int[] cols)` Sets the indicies of the datasets to display (`null` means all).
`void`	`setFoldColumn(int newFoldColumn)` Set the value of FoldColumn.
`void`	`setInstances(Instances newInstances)` Set the value of Instances.
`void`	`setOptions(String[] options)` Parses a given list of options.
`void`	`setResultMatrix(ResultMatrix matrix)` Sets the matrix to use to produce the output.
`void`	`setResultsetKeyColumns(Range newResultsetKeyColumns)` Set the value of ResultsetKeyColumns.
`void`	`setRunColumn(int newRunColumn)` Set the value of RunColumn.
`void`	`setShowStdDevs(boolean s)` Set whether standard deviations are displayed or not.
`void`	`setSignificanceLevel(double newSignificanceLevel)` Set the value of SignificanceLevel.
`void`	`setSortColumn(int newSortColumn)` Set the column to sort on, -1 means the default sorting.
`protected String`	`templateString(Instance template)` Returns a string descriptive of the key column values for the "datasets

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - m_Instances
```
protected Instances m_Instances
```
    The set of instances we will analyse
  - m_RunColumn
```
protected int m_RunColumn
```
    The index of the column containing the run number
  - m_RunColumnSet
```
protected int m_RunColumnSet
```
    The option setting for the run number column (-1 means last)
  - m_FoldColumn
```
protected int m_FoldColumn
```
    The option setting for the fold number column (-1 means none)
  - m_SortColumn
```
protected int m_SortColumn
```
    The column to sort on (-1 means default sorting)
  - m_SortOrder
```
protected int[] m_SortOrder
```
    The sorting of the datasets (according to the sort column)
  - m_ColOrder
```
protected int[] m_ColOrder
```
    The sorting of the columns (test base is always first)
  - m_SignificanceLevel
```
protected double m_SignificanceLevel
```
    The significance level for comparisons
  - m_DatasetKeyColumnsRange
```
protected Range m_DatasetKeyColumnsRange
```
    The range of columns that specify a unique "dataset" (eg: scheme plus configuration)
  - m_DatasetKeyColumns
```
protected int[] m_DatasetKeyColumns
```
    An array containing the indexes of just the selected columns
  - m_DatasetSpecifiers
```
protected PairedTTester.DatasetSpecifiers m_DatasetSpecifiers
```
    The list of dataset specifiers
  - m_ResultsetKeyColumnsRange
```
protected Range m_ResultsetKeyColumnsRange
```
    The range of columns that specify a unique result set (eg: scheme plus configuration)
  - m_ResultsetKeyColumns
```
protected int[] m_ResultsetKeyColumns
```
    An array containing the indexes of just the selected columns
  - m_DisplayedResultsets
```
protected int[] m_DisplayedResultsets
```
    An array containing the indexes of the datasets to display
  - m_Resultsets
```
protected FastVector m_Resultsets
```
    Stores a vector for each resultset holding all instances in each set
  - m_ResultsetsValid
```
protected boolean m_ResultsetsValid
```
    Indicates whether the instances have been partitioned
  - m_ShowStdDevs
```
protected boolean m_ShowStdDevs
```
    Indicates whether standard deviations should be displayed
  - m_ResultMatrix
```
protected ResultMatrix m_ResultMatrix
```
    the instance of the class to produce the output.
- Constructor Detail
  - PairedTTester
```
public PairedTTester()
```
- Method Detail
  - templateString
```
protected String templateString(Instance template)
```
    Returns a string descriptive of the key column values for the "datasets
    
    Parameters:
    
    template - the template
    
    Returns:
    
    a value of type 'String'
  - setResultMatrix
```
public void setResultMatrix(ResultMatrix matrix)
```
    Sets the matrix to use to produce the output.
    
    Specified by:
    
    setResultMatrix in interface Tester
    
    Parameters:
    
    matrix - the instance to use to produce the output
    
    See Also:
    
    ResultMatrix
  - getResultMatrix
```
public ResultMatrix getResultMatrix()
```
    Gets the instance that produces the output.
    
    Specified by:
    
    getResultMatrix in interface Tester
    
    Returns:
    
    the instance to produce the output
  - setShowStdDevs
```
public void setShowStdDevs(boolean s)
```
    Set whether standard deviations are displayed or not.
    
    Specified by:
    
    setShowStdDevs in interface Tester
    
    Parameters:
    
    s - true if standard deviations are to be displayed
  - getShowStdDevs
```
public boolean getShowStdDevs()
```
    Returns true if standard deviations have been requested.
    
    Specified by:
    
    getShowStdDevs in interface Tester
    
    Returns:
    
    true if standard deviations are to be displayed.
  - prepareData
```
protected void prepareData()
                    throws Exception
```
    Separates the instances into resultsets and by dataset/run.
    
    Throws:
    
    Exception - if the TTest parameters have not been set.
  - getNumDatasets
```
public int getNumDatasets()
```
    Gets the number of datasets in the resultsets
    
    Specified by:
    
    getNumDatasets in interface Tester
    
    Returns:
    
    the number of datasets in the resultsets
  - getNumResultsets
```
public int getNumResultsets()
```
    Gets the number of resultsets in the data.
    
    Specified by:
    
    getNumResultsets in interface Tester
    
    Returns:
    
    the number of resultsets in the data
  - getResultsetName
```
public String getResultsetName(int index)
```
    Gets a string descriptive of the specified resultset.
    
    Specified by:
    
    getResultsetName in interface Tester
    
    Parameters:
    
    index - the index of the resultset
    
    Returns:
    
    a descriptive string for the resultset
  - displayResultset
```
public boolean displayResultset(int index)
```
    Checks whether the resultset with the given index shall be displayed.
    
    Specified by:
    
    displayResultset in interface Tester
    
    Parameters:
    
    index - the index of the resultset to check whether it shall be displayed
    
    Returns:
    
    whether the specified resultset is displayed
  - calculateStatistics
```
public PairedStats calculateStatistics(Instance datasetSpecifier,
                                       int resultset1Index,
                                       int resultset2Index,
                                       int comparisonColumn)
                                throws Exception
```
    Computes a paired t-test comparison for a specified dataset between two resultsets.
    
    Specified by:
    
    calculateStatistics in interface Tester
    
    Parameters:
    
    datasetSpecifier - the dataset specifier
    
    resultset1Index - the index of the first resultset
    
    resultset2Index - the index of the second resultset
    
    comparisonColumn - the column containing values to compare
    
    Returns:
    
    the results of the paired comparison
    
    Throws:
    
    Exception - if an error occurs
  - resultsetKey
```
public String resultsetKey()
```
    Creates a key that maps resultset numbers to their descriptions.
    
    Specified by:
    
    resultsetKey in interface Tester
    
    Returns:
    
    a value of type 'String'
  - header
```
public String header(int comparisonColumn)
```
    Creates a "header" string describing the current resultsets.
    
    Specified by:
    
    header in interface Tester
    
    Parameters:
    
    comparisonColumn - a value of type 'int'
    
    Returns:
    
    a value of type 'String'
  - multiResultsetWins
```
public int[][] multiResultsetWins(int comparisonColumn,
                                  int[][] nonSigWin)
                           throws Exception
```
    Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other.
    
    Specified by:
    
    multiResultsetWins in interface Tester
    
    Parameters:
    
    comparisonColumn - the index of the comparison column
    
    nonSigWin - for storing the non-significant wins
    
    Returns:
    
    a 2d array where element [i][j] is the number of times resultset j performed significantly better than resultset i.
    
    Throws:
    
    Exception - if an error occurs
  - initResultMatrix
```
protected void initResultMatrix()
```
    clears the content and fills the column and row names according to the given sorting
  - multiResultsetSummary
```
public String multiResultsetSummary(int comparisonColumn)
                             throws Exception
```
    Carries out a comparison between all resultsets, counting the number of datsets where one resultset outperforms the other. The results are summarized in a table.
    
    Specified by:
    
    multiResultsetSummary in interface Tester
    
    Parameters:
    
    comparisonColumn - the index of the comparison column
    
    Returns:
    
    the results in a string
    
    Throws:
    
    Exception - if an error occurs
  - multiResultsetRanking
```
public String multiResultsetRanking(int comparisonColumn)
                             throws Exception
```
    returns a ranking of the resultsets
    
    Specified by:
    
    multiResultsetRanking in interface Tester
    
    Parameters:
    
    comparisonColumn - the column to compare with
    
    Returns:
    
    the ranking
    
    Throws:
    
    Exception - if something goes wrong
  - multiResultsetFull
```
public String multiResultsetFull(int baseResultset,
                                 int comparisonColumn)
                          throws Exception
```
    Creates a comparison table where a base resultset is compared to the other resultsets. Results are presented for every dataset.
    
    Specified by:
    
    multiResultsetFull in interface Tester
    
    Parameters:
    
    baseResultset - the index of the base resultset
    
    comparisonColumn - the index of the column to compare over
    
    Returns:
    
    the comparison table string
    
    Throws:
    
    Exception - if an error occurs
  - listOptions
```
public Enumeration listOptions()
```
    Lists options understood by this object.
    
    Specified by:
    
    listOptions in interface OptionHandler
    
    Returns:
    
    an enumeration of Options.
  - setOptions
```
public void setOptions(String[] options)
                throws Exception
```
    Parses a given list of options.
    Valid options are:
```
 -D <index,index2-index4,...>
  Specify list of columns that specify a unique
  dataset.
  First and last are valid indexes. (default none)
```
```
 -R <index>
  Set the index of the column containing the run number
```
```
 -F <index>
  Set the index of the column containing the fold number
```
```
 -G <index1,index2-index4,...>
  Specify list of columns that specify a unique
  'result generator' (eg: classifier name and options).
  First and last are valid indexes. (default none)
```
```
 -S <significance level>
  Set the significance level for comparisons (default 0.05)
```
```
 -V
  Show standard deviations
```
```
 -L
  Produce table comparisons in Latex table format
```
```
 -csv
  Produce table comparisons in CSV table format
```
```
 -html
  Produce table comparisons in HTML table format
```
```
 -significance
  Produce table comparisons with only the significance values
```
```
 -gnuplot
  Produce table comparisons output suitable for GNUPlot
```
    Specified by:
    
    setOptions in interface OptionHandler
    
    Parameters:
    
    options - an array containing options to set.
    
    Throws:
    
    Exception - if invalid options are given
  - getOptions
```
public String[] getOptions()
```
    Gets current settings of the PairedTTester.
    
    Specified by:
    
    getOptions in interface OptionHandler
    
    Returns:
    
    an array of strings containing current options.
  - getResultsetKeyColumns
```
public Range getResultsetKeyColumns()
```
    Get the value of ResultsetKeyColumns.
    
    Specified by:
    
    getResultsetKeyColumns in interface Tester
    
    Returns:
    
    Value of ResultsetKeyColumns.
  - setResultsetKeyColumns
```
public void setResultsetKeyColumns(Range newResultsetKeyColumns)
```
    Set the value of ResultsetKeyColumns.
    
    Specified by:
    
    setResultsetKeyColumns in interface Tester
    
    Parameters:
    
    newResultsetKeyColumns - Value to assign to ResultsetKeyColumns.
  - getDisplayedResultsets
```
public int[] getDisplayedResultsets()
```
    Gets the indices of the the datasets that are displayed (if null then all are displayed). The base is always displayed.
    
    Specified by:
    
    getDisplayedResultsets in interface Tester
    
    Returns:
    
    the indices of the datasets to display
  - setDisplayedResultsets
```
public void setDisplayedResultsets(int[] cols)
```
    Sets the indicies of the datasets to display (null means all). The base is always displayed.
    
    Specified by:
    
    setDisplayedResultsets in interface Tester
    
    Parameters:
    
    cols - the indices of the datasets to display
  - getSignificanceLevel
```
public double getSignificanceLevel()
```
    Get the value of SignificanceLevel.
    
    Specified by:
    
    getSignificanceLevel in interface Tester
    
    Returns:
    
    Value of SignificanceLevel.
  - setSignificanceLevel
```
public void setSignificanceLevel(double newSignificanceLevel)
```
    Set the value of SignificanceLevel.
    
    Specified by:
    
    setSignificanceLevel in interface Tester
    
    Parameters:
    
    newSignificanceLevel - Value to assign to SignificanceLevel.
  - getDatasetKeyColumns
```
public Range getDatasetKeyColumns()
```
    Get the value of DatasetKeyColumns.
    
    Specified by:
    
    getDatasetKeyColumns in interface Tester
    
    Returns:
    
    Value of DatasetKeyColumns.
  - setDatasetKeyColumns
```
public void setDatasetKeyColumns(Range newDatasetKeyColumns)
```
    Set the value of DatasetKeyColumns.
    
    Specified by:
    
    setDatasetKeyColumns in interface Tester
    
    Parameters:
    
    newDatasetKeyColumns - Value to assign to DatasetKeyColumns.
  - getRunColumn
```
public int getRunColumn()
```
    Get the value of RunColumn.
    
    Specified by:
    
    getRunColumn in interface Tester
    
    Returns:
    
    Value of RunColumn.
  - setRunColumn
```
public void setRunColumn(int newRunColumn)
```
    Set the value of RunColumn.
    
    Specified by:
    
    setRunColumn in interface Tester
    
    Parameters:
    
    newRunColumn - Value to assign to RunColumn.
  - getFoldColumn
```
public int getFoldColumn()
```
    Get the value of FoldColumn.
    
    Specified by:
    
    getFoldColumn in interface Tester
    
    Returns:
    
    Value of FoldColumn.
  - setFoldColumn
```
public void setFoldColumn(int newFoldColumn)
```
    Set the value of FoldColumn.
    
    Specified by:
    
    setFoldColumn in interface Tester
    
    Parameters:
    
    newFoldColumn - Value to assign to FoldColumn.
  - getSortColumnName
```
public String getSortColumnName()
```
    Returns the name of the column to sort on.
    
    Specified by:
    
    getSortColumnName in interface Tester
    
    Returns:
    
    the name of the column to sort on.
  - getSortColumn
```
public int getSortColumn()
```
    Returns the column to sort on, -1 means the default sorting.
    
    Specified by:
    
    getSortColumn in interface Tester
    
    Returns:
    
    the column to sort on.
  - setSortColumn
```
public void setSortColumn(int newSortColumn)
```
    Set the column to sort on, -1 means the default sorting.
    
    Specified by:
    
    setSortColumn in interface Tester
    
    Parameters:
    
    newSortColumn - the new sort column.
  - getInstances
```
public Instances getInstances()
```
    Get the value of Instances.
    
    Specified by:
    
    getInstances in interface Tester
    
    Returns:
    
    Value of Instances.
  - setInstances
```
public void setInstances(Instances newInstances)
```
    Set the value of Instances.
    
    Specified by:
    
    setInstances in interface Tester
    
    Parameters:
    
    newInstances - Value to assign to Instances.
  - assign
```
public void assign(Tester tester)
```
    retrieves all the settings from the given Tester
    
    Specified by:
    
    assign in interface Tester
    
    Parameters:
    
    tester - the Tester to get the settings from
  - getToolTipText
```
public String getToolTipText()
```
    returns a string that is displayed as tooltip on the "perform test" button in the experimenter
    
    Specified by:
    
    getToolTipText in interface Tester
    
    Returns:
    
    the tool tip
  - getDisplayName
```
public String getDisplayName()
```
    returns the name of the tester
    
    Specified by:
    
    getDisplayName in interface Tester
    
    Returns:
    
    the display name
  - getRevision
```
public String getRevision()
```
    Returns the revision string.
    
    Specified by:
    
    getRevision in interface RevisionHandler
    
    Returns:
    
    the revision
  - main
```
public static void main(String[] args)
```
    Test the class from the command line.
    
    Parameters:
    
    args - contains options for the instance ttests

Class PairedTTester

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

m_Instances

m_RunColumn

m_RunColumnSet

m_FoldColumn

m_SortColumn

m_SortOrder

m_ColOrder

m_SignificanceLevel

m_DatasetKeyColumnsRange

m_DatasetKeyColumns

m_DatasetSpecifiers

m_ResultsetKeyColumnsRange

m_ResultsetKeyColumns

m_DisplayedResultsets

m_Resultsets

m_ResultsetsValid

m_ShowStdDevs

m_ResultMatrix

Constructor Detail

PairedTTester

Method Detail

templateString

setResultMatrix

getResultMatrix

setShowStdDevs

getShowStdDevs

prepareData

getNumDatasets

getNumResultsets

getResultsetName

displayResultset

calculateStatistics

resultsetKey

header

multiResultsetWins

initResultMatrix

multiResultsetSummary

multiResultsetRanking

multiResultsetFull

listOptions

setOptions

getOptions

getResultsetKeyColumns

setResultsetKeyColumns

getDisplayedResultsets

setDisplayedResultsets

getSignificanceLevel

setSignificanceLevel

getDatasetKeyColumns

setDatasetKeyColumns

getRunColumn

setRunColumn

getFoldColumn

setFoldColumn

getSortColumnName

getSortColumn

setSortColumn

getInstances

setInstances

assign

getToolTipText

getDisplayName

getRevision

main