Requirements
- Essential
- Matlab R2007a (or newer)
- Matlab Toolboxes: Optimization, Statistics
- to view list of PTCS Matlab licenses, click here
- Data from /home/gbtlogs organized in the typical directory structure
- Optional
- Matlab Database Toolbox (for communication with Access)
- Microsoft XP. May need the maximum supported memory (4 Gb).
- Microsoft Access (Jet version 4 -- Access 2000/XP)
Outline of Process
- Run prepoint
- Run peak/Focus fits in Matlab
- Database selection / python concatenation into final CSV file
- Run model fit in Matlab
Note that the astronomy data fitting is compute intensive, and requires about 3 hours to run on a Dell Precision 690 with max memory with an input of six ~ 8 hour pointing runs. Lots of stuff dumps to the command window so you can see progress. Changing parameters to speed this up, e.g., the number of bootstraps per peak or parameters for the L-M optimization, is not recommended. If you want, I can investigate an alternative approach that will reduce compute time by about x10 or more.
Instructions using Microsoft Access
- Run prepoint to generate the following data files for all projects
- Astronomy
- Weather
- Refraction
- Structural Temperatures
- Air Temperatures
- Import these data into Access. See example database.
- Construct/update the Obstype table (Used to narrow queries).
- Run Matlab script mainFitAstroData. May be need to add to Matlab path variables.
- It is best to create local copies of only the files you need from /home/gbtlogs/AntennaCharacterization-StructureTemperature-StructureTemp and -AirTemp, otherwise you get 'Out of Memory' errors from Matlab when injesting lots of files you don't need.
- Currently, the script is located at /export/home/serenity/thunter/gbt/pointing/10_28_07/ModelScript/
- Import into Access as FitData by doing an update, i.e., add the data to the exisiting FitData.
- Construct track map data from 360 deg track scan and inclinometer data using mainTrackMap.m. Put the output into a Matlab table called trackMap.mat.
- Update the ObsType and RefractionError tables.
- ObsType can be obtained from the astronomy prepoint file
- RefractionError should be set to 1 for data in the range 03Sept2007 through 20Sep2007, and 0 otherwise.
- Run a make-table query in Access to select the relevant data (e.g., QueryNewTrackAzElAllData).
- Prepare to run mainGenerateModels
- (Database Toolbox). Create system DSN in XP that points at the database. Edit mainGenerateModels to use the DSN and table. b. (No Database Toolbox). Export the table created in 6 from Access to Excel, then save the Excel file as CSV. Edit_mainGenerateModels_ to use the appropriate reader function and file.
- Make a model: Run mainGenerateModels to create a model.
- Set the hour range in the top of the file (e.g. [0,24])
- Set the desired elevation range (line 18).
- Set the input file names and directory name
- Test a model: Run mainTestModels to test a model (in the workspace) on alternative data. Note that this implmentation uses the l1,l2,and l3 residual tables generated previously rahter than using the model fitted terms (which determine scale factors for the residual tables).
Instructions using Frank's python script (instead of Access)
- Run prepoint to generate the following data files for all projects
- Astronomy
- Weather
- Refraction
- Structural Temperatures
- Air Temperatures
- Run Matlab script mainFitAstroData. May be need to add to Matlab path variables.
- It is best to create local copies of only the files you need from /home/gbtlogs/AntennaCharacterization-StructureTemperature-StructureTemp and -AirTemp, otherwise you get 'Out of Memory' errors from Matlab when injesting lots of files you don't need.
- Currently, the script is located at /export/home/serenity/thunter/gbt/pointing/10_28_07/ModelScript/
- Construct track map data from a recent 360 deg track scan and inclinometer data using mainTrackMap.m. Put the output into a Matlab table called trackMap.mat.
- Currently, the script is located at /export/home/serenity/thunter/gbt/pointing/10_28_07/ModelScript/trackMap/
- Further details on running it are described here
- Run Frank's python script which combines the CSV file output from step 1 with the prepoint information, and generates a new CSV file.
- RefractionError should be set to 1 for data in the range 03Sept2007 through 20Sep2007, and 0 otherwise.
- Prepare to run mainGenerateModels
- Edit_mainGenerateModels_ to use the appropriate reader function (CSV) and file. b. Note: It was necessary to modify readDataSet.m so that fields with E- or E+ notation would be interpreted as numbers rather than strings.
- Make a model: Run mainGenerateModels to create a model.
- Set the hour range in the top of the file (e.g. [0,24])
- Set the desired elevation range (line 18).
- Set the input file names and directory name
- Test a model: Run mainTestModels to test a model (in the workspace) on alternative data. Note that this implmentation uses the l1,l2,and l3 residual tables generated previously rather than using the model fitted terms (which determine scale factors for the residual tables).
Results and Interpretations
Various well labeled graphs.
Look for systematic errors in the graphs, deviations from the univariate error distributions, and 68th percentile errors.
Subsets of the az.m, el.m, and focus.m arrays in the relevant structures. Note that the track model terms (associated with dZetaXX) in the az.m (look at az.nams for the names of the elements) are
not used directly, rather the table consisting of
track.residTables.az, .l1, .l2, and .l3 are used in the M&C implementation.
Gotchas
Database
- Need to keep the ObsType table up-to-date with project IDs and run types with associated scan ranges.
- Need to manually add entries in the RefractionError table- Project ID and either 1 (in error) or 0 (no error). This is to patch the M&C refraction error in some of the 0709xx runs. It should be zero from here on out....
- If you use really old data, note the TempSignErrors table which indicates datasets where the structural temp data was inadvertenly recorded by M&C as abs(temp).
--
KimConstantikes - 04 Aug 2007
Linux
Columns in final CSV file
1 |
ProjectID |
2 |
ObsType |
3 |
Type |
4 |
Pol |
5 |
AstType |
6 |
Scan |
7 |
MJDate |
8 |
Source |
9 |
PointingModel |
10 |
J2000RA |
11 |
J2000Dec |
12 |
ObsFreq |
13 |
Obsc_Az |
14 |
Obsc_El |
15 |
Mnt_Az |
16 |
Mnt_El |
17 |
Peak |
18 |
PeakError |
19 |
Width |
20 |
WidthError |
21 |
SNR |
22 |
Error |
23 |
Value |
24 |
BaselineRMS |
25 |
AppRefModelVal |
26 |
NewRefModelVal |
27 |
RefractError |
28 |
WindVel |
29 |
WindDir |
30 |
gbtts1_2001 |
31 |
gbtts1_2005 |
32 |
gbtts2_2001 |
33 |
gbtts2_2002 |
34 |
gbtts2_2004 |
35 |
gbtts3_2001 |
36 |
gbtts3_2002 |
37 |
gbtts3_2003 |
38 |
gbtts3_2004 |
39 |
gbtts4_2001 |
40 |
gbtts4_2002 |
41 |
gbtts4_2003 |
42 |
gbtts4_2004 |
43 |
gbtts4_2005 |
44 |
gbtts4_2006 |
45 |
gbtts5_2001 |
46 |
gbtts5_2002 |
47 |
gbtts5_2003 |
48 |
gbtts5_2004 |
49 |
at1 |
50 |
at5 |
51 |
at2 |
52 |
at3 |
53 |
at4 |
54 |
SR_XP |
55 |
SR_XPS |
56 |
SR_YP |
57 |
SR_YPS |
58 |
SR_ZP |
59 |
SR_ZPS |
60 |
SR_XT |
61 |
SR_XTS |
62 |
SR_YT |
63 |
SR_YTS |
64 |
SR_ZT |
65 |
SR_ZTS |
66 |
SMNTC_AZ |
67 |
SMNTC_EL |
68 |
dT1Fdt |
69 |
dT2Fdt |
70 |
dT3Fdt |
71 |
dT4Fdt |
72 |
dT5Fdt |
73 |
dT6Fdt |
74 |
dT1Adt |
75 |
dT2Adt |
76 |
dT3Adt |
77 |
dT4Adt |
78 |
dT1Edt |
79 |
dT2Edt |
80 |
dT3Edt |
81 |
dT4Edt |
--
ToddHunter - 27 Jan 2008