Requirements

  • Essential
    • Matlab R2007a (or newer)
    • Matlab Toolboxes: Optimization, Statistics
    • to view list of PTCS Matlab licenses, click here
    • Data from /home/gbtlogs organized in the typical directory structure
  • Optional
    • Matlab Database Toolbox (for communication with Access)
    • Microsoft XP. May need the maximum supported memory (4 Gb).
    • Microsoft Access (Jet version 4 -- Access 2000/XP)

Outline of Process

  • Run prepoint
  • Run peak/Focus fits in Matlab
  • Database selection / python concatenation into final CSV file
  • Run model fit in Matlab

Note that the astronomy data fitting is compute intensive, and requires about 3 hours to run on a Dell Precision 690 with max memory with an input of six ~ 8 hour pointing runs. Lots of stuff dumps to the command window so you can see progress. Changing parameters to speed this up, e.g., the number of bootstraps per peak or parameters for the L-M optimization, is not recommended. If you want, I can investigate an alternative approach that will reduce compute time by about x10 or more.

Instructions using Microsoft Access

  1. Run prepoint to generate the following data files for all projects
    • Astronomy
    • Weather
    • Refraction
    • Structural Temperatures
    • Air Temperatures
  2. Import these data into Access. See example database.
  3. Construct/update the Obstype table (Used to narrow queries).
  4. Run Matlab script mainFitAstroData. May be need to add to Matlab path variables.
    • It is best to create local copies of only the files you need from /home/gbtlogs/AntennaCharacterization-StructureTemperature-StructureTemp and -AirTemp, otherwise you get 'Out of Memory' errors from Matlab when injesting lots of files you don't need.
    • Currently, the script is located at /export/home/serenity/thunter/gbt/pointing/10_28_07/ModelScript/
  5. Import into Access as FitData by doing an update, i.e., add the data to the exisiting FitData.
  6. Construct track map data from 360 deg track scan and inclinometer data using mainTrackMap.m. Put the output into a Matlab table called trackMap.mat.
  7. Update the ObsType and RefractionError tables.
    • ObsType can be obtained from the astronomy prepoint file
    • RefractionError should be set to 1 for data in the range 03Sept2007 through 20Sep2007, and 0 otherwise.
  8. Run a make-table query in Access to select the relevant data (e.g., QueryNewTrackAzElAllData).
  9. Prepare to run mainGenerateModels
    1. (Database Toolbox). Create system DSN in XP that points at the database. Edit mainGenerateModels to use the DSN and table. b. (No Database Toolbox). Export the table created in 6 from Access to Excel, then save the Excel file as CSV. Edit_mainGenerateModels_ to use the appropriate reader function and file.
  10. Make a model: Run mainGenerateModels to create a model.
    • Set the hour range in the top of the file (e.g. [0,24])
    • Set the desired elevation range (line 18).
    • Set the input file names and directory name
  11. Test a model: Run mainTestModels to test a model (in the workspace) on alternative data. Note that this implmentation uses the l1,l2,and l3 residual tables generated previously rahter than using the model fitted terms (which determine scale factors for the residual tables).

Instructions using Frank's python script (instead of Access)

  1. Run prepoint to generate the following data files for all projects
    • Astronomy
    • Weather
    • Refraction
    • Structural Temperatures
    • Air Temperatures
  2. Run Matlab script mainFitAstroData. May be need to add to Matlab path variables.
    • It is best to create local copies of only the files you need from /home/gbtlogs/AntennaCharacterization-StructureTemperature-StructureTemp and -AirTemp, otherwise you get 'Out of Memory' errors from Matlab when injesting lots of files you don't need.
    • Currently, the script is located at /export/home/serenity/thunter/gbt/pointing/10_28_07/ModelScript/
  3. Construct track map data from a recent 360 deg track scan and inclinometer data using mainTrackMap.m. Put the output into a Matlab table called trackMap.mat.
    • Currently, the script is located at /export/home/serenity/thunter/gbt/pointing/10_28_07/ModelScript/trackMap/
    • Further details on running it are described here
  4. Run Frank's python script which combines the CSV file output from step 1 with the prepoint information, and generates a new CSV file.
    • RefractionError should be set to 1 for data in the range 03Sept2007 through 20Sep2007, and 0 otherwise.
  5. Prepare to run mainGenerateModels
    1. Edit_mainGenerateModels_ to use the appropriate reader function (CSV) and file. b. Note: It was necessary to modify readDataSet.m so that fields with E- or E+ notation would be interpreted as numbers rather than strings.
  6. Make a model: Run mainGenerateModels to create a model.
    • Set the hour range in the top of the file (e.g. [0,24])
    • Set the desired elevation range (line 18).
    • Set the input file names and directory name
  7. Test a model: Run mainTestModels to test a model (in the workspace) on alternative data. Note that this implmentation uses the l1,l2,and l3 residual tables generated previously rather than using the model fitted terms (which determine scale factors for the residual tables).

Results and Interpretations

  • Graphical Output
Various well labeled graphs.
  • Performance Assessment
Look for systematic errors in the graphs, deviations from the univariate error distributions, and 68th percentile errors.
  • Model Parameters
Subsets of the az.m, el.m, and focus.m arrays in the relevant structures. Note that the track model terms (associated with dZetaXX) in the az.m (look at az.nams for the names of the elements) are not used directly, rather the table consisting of track.residTables.az, .l1, .l2, and .l3 are used in the M&C implementation.

Gotchas

Database

  • Need to keep the ObsType table up-to-date with project IDs and run types with associated scan ranges.
  • Need to manually add entries in the RefractionError table- Project ID and either 1 (in error) or 0 (no error). This is to patch the M&C refraction error in some of the 0709xx runs. It should be zero from here on out....
  • If you use really old data, note the TempSignErrors table which indicates datasets where the structural temp data was inadvertenly recorded by M&C as abs(temp).

-- KimConstantikes - 04 Aug 2007

Linux

Columns in final CSV file

1 ProjectID
2 ObsType
3 Type
4 Pol
5 AstType
6 Scan
7 MJDate
8 Source
9 PointingModel
10 J2000RA
11 J2000Dec
12 ObsFreq
13 Obsc_Az
14 Obsc_El
15 Mnt_Az
16 Mnt_El
17 Peak
18 PeakError
19 Width
20 WidthError
21 SNR
22 Error
23 Value
24 BaselineRMS
25 AppRefModelVal
26 NewRefModelVal
27 RefractError
28 WindVel
29 WindDir
30 gbtts1_2001
31 gbtts1_2005
32 gbtts2_2001
33 gbtts2_2002
34 gbtts2_2004
35 gbtts3_2001
36 gbtts3_2002
37 gbtts3_2003
38 gbtts3_2004
39 gbtts4_2001
40 gbtts4_2002
41 gbtts4_2003
42 gbtts4_2004
43 gbtts4_2005
44 gbtts4_2006
45 gbtts5_2001
46 gbtts5_2002
47 gbtts5_2003
48 gbtts5_2004
49 at1
50 at5
51 at2
52 at3
53 at4
54 SR_XP
55 SR_XPS
56 SR_YP
57 SR_YPS
58 SR_ZP
59 SR_ZPS
60 SR_XT
61 SR_XTS
62 SR_YT
63 SR_YTS
64 SR_ZT
65 SR_ZTS
66 SMNTC_AZ
67 SMNTC_EL
68 dT1Fdt
69 dT2Fdt
70 dT3Fdt
71 dT4Fdt
72 dT5Fdt
73 dT6Fdt
74 dT1Adt
75 dT2Adt
76 dT3Adt
77 dT4Adt
78 dT1Edt
79 dT2Edt
80 dT3Edt
81 dT4Edt

-- ToddHunter - 27 Jan 2008
Topic attachments
I Attachment Action Size Date Who Comment
ModelScript.tartar ModelScript.tar manage 180 K 2008-01-29 - 14:21 ToddHunter tar file of Matlab scripts, as of Jan 23, 2008
avgStructuralTemp.pngpng avgStructuralTemp.png manage 3 K 2008-02-15 - 13:56 ToddHunter histogram of avg structure temp of Sep2007 X-band
Topic revision: r24 - 2010-09-10, FrankGhigo
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback