JSOC Help and Examples


Examples of JSOC RecordSet Specifiers

Basics

If you are familiar with the basic structure of JSOC Datasets including the JSOC term "PrimeKeyword" please review The JSOC data series in the JSOC Wiki.

Simple time series examples

Many dataseries are a time ordered sequence of records, such as the basic set of HMI magnetograms observed one for each 45-seconds. For such series the first PrimeKey is the time of the record. Its name will usually be "T_REC" for HMI and MDI data but may be some other time keyword such as "T_OBS". The name of a PrimeKey need not be specified if you know the order in which they are defined. The standard is that a dataseries which is basically a time series, the first PrimeKey will the the ordering time keyword. The difference between the two samples above is that the first form, a range of times, includes any records that exist for the the full range including both end times. In the second case, where the "/30m" can be read as "over 30 minutes", the span of records matching will stop just before the given time plus the span duration. This makes sense if you want all the images for May 1, 2010 then you would write: "hmi.M_45s[2010.05.01/1d]" and the result would include only magnetograms for May 1, not the first one for May 2. Of course you could specify "hmi.M_45s[2010.05.01_00:00:00-2010.05.01_23:59:15]" but the compact form is both easier to type and easier to read. The first sample above returns 41 records and the second returns 40.

Note about notation for time

The JSOC DRMS time parsing code can recognize several formats for expressing time. In particular it understands a subset of ISO-8601 formats such as 2010-07-25T13:30:00 and the SDO/HMI (and SOHO/MDI) format of 2010.07.25_13:30_00. Trailing time or date parts that are omitted default to the start of the respecitve interval. E.g. 2010.07.25_13 corresponds to 13:00 UTC on 25 July 2010 as one would expect. If the ISO format is used, at least the hour part must be present so the programs can tell the difference between an ISO time and a range of times. See JSOC-TN 07-001 for the full specification.

Time zones are a special topic. HMI data it time tagged in TAI time, "International Atomic Time", since they are observed on SDO using TAI time. There are no leap-seconds in TAI time as there are in UTC so for long time series such as the 14 year SOHO/MDI set of heliosiesmology data one does not need to worry about the 5 leap-seconds that were inserted during that time. All SDO data are intrinsically observed using TAI. For the convenience of people the times can also be expressed in UTC. HMI data has "nicer" time stamps when expressed in TAI. By choice, AIA uses UTC for most user-level time stamps. Either may be used for any JSOC data requests. Since the offset, now 34 seconds, is more than half the basic HMI obseving cadence of 45 seconds, you will often not get the record you expect if you use UTC to ask for HMI data. The default for an omitted time zone specifier is UTC. Thus if you ask for hmi_test.M_45s[2010.07.25] you will get the magnetogram labelled hmi_test.M_45s[2010.07.25_00:00:45_TAI].

The rule of thumb is: Use TAI when specifing HMI or MDI data but use UTC for AIA.

More about Prime Keys

A JSOC Dataseries may have multiple Prime Keys, between 0 and 15. Usually 1 or a few. When multiple Prime Keys are used, a record is fully specified by the complete set of keyword values for the PrimeKeys. The PrimeKeys can be thought of as a filter acting on the full set of records in the given dataseries. I.e. only records matching all of the given PrimeKey values will be selected. If some PrimeKeys are omitted then all records matching the given PrimeKeys but having any values for the omitted PrimeKeys will be returned.

The full set of PrimeKeys for each series are listed in lookdata both on the "Series Content" tab and at the top of the "RecordSet Select" tab. If you check the checkbox labelled "Check box to show the QueryBuilder" located just under the title "Select Records and Get Record Count" on the "RecordSet Select" tab, lookdata will provide you with an expanded template for building a recordset query using all of the available PrimeKeys.

Example 2:

  • Query Builder for hmi_test.M_45s
  • This example shows what you see with the checkbox checked and the hmi_test.M_45s series selected. It shows that hmi_test.M_45s has two PrimeKeys, "T_REC" and "CAMERA". HMI has two cameras, one called 1:Side and the other 2:Front (referring to the location of the electronics box on the HMI instrument). Since each basic observable such as magnetic field or Doppler velocity can be computed for data taken by either camera, we indicate the particular camera with the CAMERA PrimeKey. At least up through July 2010, all 45-second cadence Magnetograms ahve been made using only the Front camera so we may safely omit the CAMERA keyword in a data request.

    Generally, each PrimeKey will be in some sense orthogonal to the others. They specify different quantities which are all needed for a unique record specification. This is not required. In fact, AIA and HMI each use both T_OBS and FSN as PrimeKeys for the "Level-1" dataseries. T_OBS is always the time of observation of a quantity. For the low-level HMI and AIA data, "FSN" is the "Frame Sequence Number" and is a unique number provided by the onboard software when each image is created. Since no two images can be made at exactly the same time, FSN is redundant with T_OBS. However, rounded to the nearest second and in testing multiple images can be obtained with the same T_OBS rounded value. But the reason we chose to use both for this low-level data is that the data is generated based on FSN but used based on T_OBS. You should use "T_OBS" for aia.lev1 data unless you really know what you are after.

    While the built-in PrimeKey formats are convenient with notation like "2010.07.25/6h" sometimes the user wants to either build a query from other not Prime Keywords or to make a query that is noyt easy to specify using the PrimeKey special logic. In those cases we provide the ability to specify one or more very geenral SQL "Where" clauses. An SQL where clause contents may be provided in one or more "[? ... ?]" or "[! ... !]" clauses. The word "where" is not needed, or allowed.