Deployment

Standard deployment

The TMC Low is packaged as a Helm chart and can be deployed using Helm commands. The default deployment configuration is assumed to be the SKA production environment. In the current version, the TMC supports two subarray operations. The following list shows the default number of instances deployed for each TMC component.

  1. Central Node - 1

  2. Subarray Node - 2

  3. CSP Master Leaf Node - 1

  4. CSP Subarray Leaf Node - 2

  5. SDP Master Leaf Node - 1

  6. SDP Subarray Leaf Node - 2

  7. MCCS Master Leaf Node - 1

  8. MCCS Subarray Leaf Node - 2

  9. Resource Monitor - 1

  10. Quality Monitor - 2

Warning

The number of instances of Central Node, MCCS Master Leaf Node, SDP Master Leaf Node, CSP Master Leaf Node and Resource Monitor should always be one even though it is technically possible to deploy multiple instances.

To deploy the TMC, use the following command in a terminal:

helm install my-tmc-release https://artefact.skao.int/repository/helm-internal/ska-tmc-low --namespace ska-tmc-low

It is possible to customize the deployment as needed. To do so, modify the values.yaml file in the TMC chart. Alternatively, use the –set option on the command line with the Helm install command.

Basic Customization options

Number of subarrays

The number of subarrays can be configured as needed. Set the variable named subarray_count to the desired value. This option affects the number of instances of the following components.

  1. Subarray Node

  2. CSP Subarray Leaf Node

  3. SDP Subarray Leaf Node

  4. MCCS Subarray Leaf Node

Note

The value of subarray_count is controlled under global section of values.yaml.

Tango host

This option allows you to specify the desired Tango facility. The variable tango_host is used to specify the Tango facility. By default, the value of this variable is set to databaseds-tango-base-test:10000.

Command timeout

The CommandTimeout attribute is introduced to allow updating the timeout value for commands without requiring a redeployment. This provides flexibility in tuning the timeout dynamically at runtime based on operational needs.

The CommandTimeOutDefault property is also introduced, which can be used to set a default timeout value during the deployment phase. This ensures that an initial timeout value is preconfigured when the component starts for the first time.

Logging

The basic setting for logging allows setting logging level of all TMC devices via TMCLoggingLevel parameter in global section. By default, the logging level is set to 4 (INFO level). To set the logging level of one or more specific components, refer to the advanced customization options.

Usage

  • CommandTimeout attribute - Can be updated at runtime without redeployment. - Helps in adapting to varying command execution times.

  • CommandTimeOutDefault property - Configurable in the deployment configuration (e.g., values.yaml). - Sets the initial timeout value at startup.

This option sets the timeout for which the TMC components wait for commands invoked on lower-level Tango devices to complete. This timeout should be set for each TMC component. To set the desired timeout value, navigate to deviceServers -> <component name> in the values.yaml file. Locate the CommandTimeOutDefault variable and set an integer value in seconds.

Warning

When setting command timeout values, it is essential to set higher timeout values for Central Node and Subarray Node than any Leaf Node. This this is because, according to the TMC architecture, the Central Node and Subarray Node are higher in the hierarchy, while Leaf Nodes are lower. The commands flow from the Central Node and Subarray Node to the Leaf Nodes and then to the subsystems. The higher level nodes need to factor in the command timeout set on lower level components.

Advanced Customization options

The following are the advanced options. These options are mainly useful for developers and AIV engineers for testing purposes. Customize these parameters done carefully.

  1. fileUsers can provide a custom device server configuration file for

    nodes. Defaults are: configuration files.

  2. enabled : Users can disable any node by setting this value to False. The default is True for all nodes.

  3. domain : It is the domain name of TMC TANGO device. The value is set to “low-tmc”.

  4. Variables under global section

    1. subarray_count : This value is present under the global section. It is the number of subarrays and is set to 1.

  5. Variables under devices_fqdn section

    1. tmc_subarray_prefix : This value is present under the global section. Users can use it to change the FQDN prefix of SubarrayNode.

    2. csp_subarray_ln_prefix : This value is present under global, User can use this to change the FQDN prefix of CspSubarrayLeafNode.

    3. sdp_subarray_ln_prefix : This value is present under global, User can use this to change the FQDN prefix of SdpSubarrayLeafNode.

    4. csp_master_ln : This value is present under global, User can use this to change the FQDN of CspMasterLeafNode.

    5. sdp_master_ln : This value is present under global, User can use this to change the FQDN of SdpMasterLeafNode.

    6. csp_subarray_prefix : This value is present under global, User can use this to change the FQDN prefix of CSP Subarray.

    7. sdp_subarray_prefix : This value is present under global, User can use this to change the FQDN prefix of SDP Subarray.

    8. csp_master : This value is present under global, User can use this to change the FQDN of CSP Master.

    9. sdp_master : This value is present under global, User can use this to change the FQDN of SDP Master.

    10. mccs_master : This value is present under global, User can use this to change the FQDN of MCCS Master.

    11. mccs_master_ln : This value is present under global, User can use this to change the FQDN of MCCS Master Leaf Node.

    12. mccs_subarray_prefix : This value is present under global, User can use this to change the FQDN prefix of MCCS Subarray.

    13. mccs_subarray_ln_prefix : This value is present under global, User can use this to change the FQDN prefix of MCCS Subarray Leaf Node.

    14. resource_monitor_prefix : This value is present under global. It can be used to change the FQDN prefix of the Resource Monitor device.

  6. subarray_devices : List of subarray-devices

  7. LoggingLevel : To set the logging level for a specific TMC component, define this key under deviceServers -> <component_name> in values.yaml.

    This is an optional parameter. If specified, it takes precedence over the value set in TMCLoggingLevel under the global section.

Component specific configuration

This section specifies the configuration options for individual TMC components. Navigate to the deviceServers.<component> section in the values.yaml file.

Central Node

  1. SkuidService : This refers to the value for SKUID service. Currently defaults to “ska-ser-skuid-test-svc.ska-tmc-centralnode.svc.techops.internal.skao.int:9870”.

  2. LivelinessCheckPeriod : This refers to the Period (in seconds) for the liveliness probe to monitor each device in a loop. Currently defaults to 0.5 seconds.

  3. EventSubscriptionCheckPeriod : This refers to the Period (in seconds) for the event subscriber to check the device subscriptions in a loop. Currently defaults to 0.5 seconds.

  4. CommandTimeOutDefault : This refers to the Timeout (in seconds) for the command execution. Currently defaults to 80 seconds.

  5. AssignResourcesInterface : This refers to the interface value of AssignResources schema. Currently defaults to “https://schema.skao.int/ska-low-tmc-assignresources/4.0”.

  6. ReleaseResourcesInterface : This refers to the interface value of ReleaseResources schema. Currently defaults to “https://schema.skao.int/ska-low-tmc-releaseresources/3.0”.

  7. family : This refers to the family name of CentralNode TANGO device. Currently defaults to “central-node”.

  8. member : This refers to the member of CentralNode TANGO device. Currently defaults to “0”.

  9. IsAutoRecoveryEnabled : This refers to the feature flag i.e used temporarily till all the nodes are tested with auto recovery functionality. Currently defaults to “true”.

  10. SubarrayPrefix : This refers to the prefix for subarray devices. Currently defaults to “low-tmc/subarray/”.

  11. DefaultArrayLayoutSourceURIs : This refers to the default array layout source URIs. Currently defaults to “”.

  12. DefaultArrayLayoutPath : This refers to the default array layout path. Currently defaults to “”.

Subarray Node

  1. CspAssignResourcesInterfaceURL : Interface version for CSP assign resources command. Currently defaults to “https://schema.skao.int/ska-low-csp-assignresources/7.5”

  2. CspScanInterfaceURL : Interface version for CSP scan command. Currently defaults to “https://schema.skao.int/ska-low-csp-scan/8.4”

  3. SdpScanInterfaceURL : Interface version for SDP scan command. Currently defaults to “https://schema.skao.int/ska-sdp-scan/2.1”

  4. MccsConfigureInterfaceURL : Interface version for MCCS configure command. Currently defaults to “https://schema.skao.int/ska-low-mccs-configure/1.0”

  5. MccsReleaseInterfaceURL : Interface version for MCCS release command. Currently defaults to “https://schema.skatelescope.org/ska-low-mccs-controller-release/2.0”

  6. MccsScanInterfaceURL : Interface version for MCCS scan command. Currently defaults to “https://schema.skao.int/ska-low-mccs-scan/3.0”

  7. JonesURI : URI for Jones Matrix. Currently defaults to “tango://jones.skao.int/low/stn-beam/1”.

  8. LivelinessCheckPeriod : This refers to the Period (in seconds) for the liveliness probe to monitor each device in a loop. Currently defaults to 0.5 seconds.

  9. EventSubscriptionCheckPeriod : This refers to the Period (in seconds) for the event subscriber to check the device subscriptions in a loop. Currently defaults to 0.5 seconds.

  10. CommandTimeOutDefault : This refers to the timeout (in seconds) for the command execution. Currently defaults to 70 seconds.

  11. AbortCommandTimeOut : This refers to the timeout for the Subarray ABORTED obsState transition. Once the AbortCommandTimeOut exceeds, SubarrayNode transitions to obsState FAULT. Currently defaults to 60 seconds.

  12. family : This refers to the family name of SubarrayNode TANGO device. Currently defaults to “subarray”.

  13. IsAutoRecoveryEnabled : This refers to the feature flag i.e used temporarily till all the nodes are tested with auto recovery functionality. Currently defaults to “true”.

  14. TelmodelSource : This refers to the TelmodelSource of SubarrayNode TANGO device. Currently defaults to “car://gitlab.com/ska-telescope/ska-telmodel-data?main#tmdata”.

  15. TelmodelPath : This refers to the TelmodelPath of SubarrayNode TANGO device. Currently defaults to “instrument/ska1_low/layout/low-layout.json”.

  16. ScanStartTimeEnabled : This refers to the property provided to enable/disable start_time in Scan JSON. Currently defaults to false.

  17. ScanDurationEnabled : This refers to the property provided to enable/disable duration in Scan JSON. Currently defaults to false.

  18. PreRecoveryCheckTimeDuration : This property refers to time duration subarray node needs to wait before evaluating the auto recovery condition. Within the time duration if the subsystems transition into final/previous observation state then the auto recovery can be proceeded as per conditions. Default value is 1 second.

    Note

    The PreRecoveryCheckTimeDuration property handles a specific edge case in the automatic recovery of the subarray. Do not modify the default value without first understanding the Subarray Node’s automatic recovery behaviour. Please refer to TMC Recovery mechanism in case of subsystem failure for more details.

SDP Subarray Leaf Node

  1. LivelinessCheckPeriod : This refers to the Period (in seconds) for the liveliness probe to monitor each device in a loop. Currently defaults to 0.5 seconds.

  2. EventSubscriptionCheckPeriod : This refers to the Period (in seconds) for the event subscriber to check the device subscriptions in a loop. Currently defaults to 0.5 seconds.

  3. CommandTimeOutDefault : This refers to the timeout (in seconds) for the command execution. Currently defaults to 50 seconds.

  4. AdapterTimeOut : This refers to the timeout (in seconds) for the adapter creation. This property is for internal use. Currently defaults to 2 seconds.

  5. family : This refers to the family name of SDP Subarray Leaf Node TANGO device. Currently defaults to “subarray-leaf-node-sdp”.

SDP Master Leaf Node

  1. LivelinessCheckPeriod : This refers to the Period (in seconds) for the liveliness probe to monitor each device in a loop. Currently defaults to 0.5 seconds.

  2. EventSubscriptionCheckPeriod : This refers to the Period (in seconds) for the event subscriber to check the device subscriptions in a loop. Currently defaults to 0.5 seconds.

  3. AdapterTimeOut : This refers to the timeout (in seconds) for the adapter creation. This property is for internal use. Currently defaults to 2 seconds.

  4. family : This refers to the family name of SDP Master Leaf Node TANGO device. Currently defaults to “leaf-node-sdp”.

  5. member : This refers to the member of SDP Master Leaf Node TANGO device. Currently defaults to “0”.

CSP Master Leaf Node

  1. LivelinessCheckPeriod : This refers to the Period (in seconds) for the liveliness probe to monitor each device in a loop. Currently defaults to 0.5 seconds.

  2. EventSubscriptionCheckPeriod : This refers to the Period (in seconds) for the event subscriber to check the device subscriptions in a loop. Currently defaults to 0.5 seconds.

  3. AdapterTimeOut : This refers to the timeout (in seconds) for the adapter creation. This property is for internal use. Currently defaults to 2 seconds.

  4. family : This refers to the family name of CSP Master Leaf Node TANGO device. Currently defaults to “leaf-node-csp”.

  5. member : This refers to the member of CSP Master Leaf Node TANGO device. Currently defaults to “0”.

CSP Subarray Leaf Node

  1. DelayCadence : This refers to the time difference (in seconds) between each publication of delay values to the delayModel attribute on the CspSubarrayLeafNode. Currently defaults to 300 seconds.

  2. DelayValidityPeriod : This represents the duration (in seconds) for which delay values remain valid after being published. Currently defaults to 600 seconds.

  3. DelayModelTimeInAdvance : This indicates the time in seconds by which delay values need to be available in advance. Currently defaults to 0 seconds.

  4. PSTDelayCadence : This refers to the time difference (in seconds) between each publication of delay values for the PST beam. Currently defaults to 300 seconds.

  5. PSTDelayValidityPeriod : This represents the duration (in seconds) for which delay values remain valid after being published. Currently defaults to 600 seconds.

  6. PSTDelayModelTimeInAdvance : This indicates the time in seconds by which delay values need to be available in advance. Currently defaults to 0 seconds.

  7. pss.delay_cadence : This refers to the time difference (in seconds) between each publication of delay values for the PSS beam. Currently defaults to 300 seconds.

  8. pss.delay_validity_period : This represents the duration (in seconds) for which delay values remain valid after being published. Currently defaults to 600 seconds.

  9. pss.delay_model_time_in_advance : This indicates the time in seconds by which delay values need to be available in advance. Currently defaults to 0 seconds.

  10. LivelinessCheckPeriod : This refers to the Period (in seconds) for the liveliness probe to monitor each device in a loop. Currently defaults to 0.5 seconds.

  11. EventSubscriptionCheckPeriod : This refers to the Period (in seconds) for the event subscriber to check the device subscriptions in a loop. Currently defaults to 0.5 seconds.

  12. CommandTimeOutDefault : This refers to the timeout (in seconds) for the command execution. Currently defaults to 50 seconds.

  13. AdapterTimeOut : This refers to the timeout (in seconds) for the adapter creation. This property is for internal use. Currently defaults to 2 seconds.

  14. family : This refers to the family name of CSP Subarray Leaf Node TANGO device. Currently defaults to “subarray-leaf-node-csp”.

MCCS Master Leaf Node

  1. LivelinessCheckPeriod : This refers to the Period (in seconds) for the liveliness probe to monitor each device in a loop. Currently defaults to 0.5 seconds.

  2. EventSubscriptionCheckPeriod : This refers to the Period (in seconds) for the event subscriber to check the device subscriptions in a loop. Currently defaults to 0.5 seconds.

  3. CommandTimeOutDefault : This refers to the timeout (in seconds) for the command execution. Currently defaults to 50 seconds.

  4. AdapterTimeOut : This refers to the timeout (in seconds) for the adapter creation. This property is for internal use. Currently defaults to 2 seconds.

  5. family : This refers to the family name of MCCS Master Leaf Node TANGO device. Currently defaults to “leaf-node-mccs”.

  6. member : This refers to the member of MCCS Master Leaf Node TANGO device. Currently defaults to “0”.

MCCS Subarray Leaf Node

  1. LivelinessCheckPeriod : This refers to the Period (in seconds) for the liveliness probe to monitor each device in a loop. Currently defaults to 0.5 seconds.

  2. EventSubscriptionCheckPeriod : This refers to the Period (in seconds) for the event subscriber to check the device subscriptions in a loop. Currently defaults to 0.5 seconds.

  3. CommandTimeOutDefault : This refers to the timeout (in seconds) for the command execution. Currently defaults to 50 seconds.

  4. AdapterTimeOut : This refers to the timeout (in seconds) for the adapter creation. This property is for internal use. Currently defaults to 2 seconds.

  5. family : This refers to the family name of MCCS Subarray Leaf Node TANGO device. Currently defaults to “subarray-leaf-node-mccs”.

Resource Monitor

  1. LivelinessCheckPeriod : This defines how often (in seconds) the system checks if each connected device is active. The default value is 0.5 seconds.

  2. EventSubscriptionCheckPeriod : This defines the period (in seconds) for the event subscriber to verify and maintain active subscriptions. Currently defaults to 0.5 seconds.

  3. family : This refers to the family name of the Resource Monitor Tango device. Currently defaults to “resource-monitor”.

Quality Monitor

  1. family : This refers to the family name of the Quality Monitor Tango device. Currently defaults to “subarray-quality-monitor”.

  2. mccs_tango_facility : This refers to the MCCS Tango facility. Currently defaults to empty string.

  3. csp_tango_facility : This refers to the CSP Tango facility. Currently defaults to empty string.

  4. sdp_tango_facility : This refers to the SDP Tango facility. Currently defaults to empty string.

  5. readyToScanMap : This refers to the dictionary that allows to specify readyToScan rule for a specific tmc quality monitor instance. The key should be “tmctransducer1”, “tmctransducer2”, …. , tmctransducer16. The value should specify the rule for example “qualityMetricsCspSubarray[‘low-cbf/subarray/01’][‘processorsReadyPercent’] == 100.0”