Kumar: found sloppy code in Feed table, will check into github.
Defined HPC parallelization development plan
Major improvements to single dish pipeline context handling, memory usage, and performance
HPC development of tier0 parallelization of plotting and calibrator imaging code
Provided standard pipeline images to ADMIIT project for testing
Fixed a bug when the user selected the "residual" column to be flagged in flagdata. (CAS-7642)
Modified partition so that the SYSPOWER table only exists in the first subMS and is linked in the other ones. This is similar to what we do for the POINTING and SYSCAL tables, to avoid having duplicates of a big table. (CAS 7662)
Working on adding support for spw selections by name in the online flags coming from importasdm. (CAS-7623).
HPC meeting and related discussions
Improvements to resolved flux calibrator heuristics
PWV plotting code refactor
Showed JAO staff how to import use caltables (antenna correction table) into pipeline context with hif_import_calstate
Cal library parser now works with strings or lists/arrays of strings
CAS-7675 resolved - time units now hh:mm:ss
Several small things
Speeding up convolution functions (chirped z calculation)
Justo González Villalba
HPC Face to Face
CARTA: Committed first of ??? versions of Grid Controls
CARTA: Miscellaneous bug fixing.
(CAS-7674) Testing of new process to commit to casacore GitHub: Read Jim's and Julian's document regarding making changes to casacore github. I thank them for their effort on this document, which provides very good step by step instructions. I added things in a few places that tripped me up, and thank them for their quick clarification of points I did not understand. Things of note, IMO:
I understand there is no way around this current process and that this is currently the shortest path to getting things into github, but I find it very daunting, partly because most of our code base remains in svn. Those of us who deal with casacore, now have to keep both svn and git commands, as well as github actions, in short term memory in order to be productive.
the github version of casacore cannot be configured using the version of cfitsio that CASA uses, which required me to build a newer version of cfitsio, and configure my github build to use that
as Dirk reported, the code/ tree will not configure against the current version of github casacore, which it deems not recent enough
as Dirk reported, after updating code/CMakeLists.txt to allow for the version of github casacore so code/ can be configured, the code/ tree will not compile against the github version of casacore
I have not yet been granted write access to the master version of casacore, so I must submit pull requests to have others commit my changes there.
GitHub will not work on standard firefox version that comes with NRAO's RHEL 5. Asked for, and was granted permission to upgrade to RHEL6. Awaiting Darrell's return in order to coordinate with him for installing necessary CASA development dependencies.
(CAS-437) Coordinate system of the antenna table after export to UV FITS: Requested confirmation if this is still an issue, since work seems to have been done on it.
(CAS-5583) Frequency in exportuvfits output: Requested dataset that seems to have disappeared.
(CAS-4852) exportuvfits writes wrong field table information when multisource=False: have fix in my sandbox, awaiting for casacore github dust to settle a bit more before trying to commit it
(CAS-7319) MSMD additions: This was the ticket I used for testing github. pull request initiated. awaiting whatever is supposed to happen next.
(CAS-7583) listobs failure on CASA 4.4. prerelease: Continued discussion of how old MSes, currently used in some examples, are invalid because they have zero rows in their STATE subtables, but their main tables have some STATE_ID values which are non-negative.
Things from previous week that were not touched on in previous MMM because of lack of time:
Met with Peter Teuben who was in CV marketing ADMIT with Mundy and Pound. Topics:
He mentioned he isi considering making the proposal that most miriad code be removed from casacore and put into a third party library on which casacore would depend. I said I wasn't sure but I suspected that proposal would please some folks.
He asked about the apparent delay in our team moving to the GitHub version of casacore. I told him I had very little information on that and pointed him to Jeff and Jim.
He asked about casacore 2, and I wasn't sure what that was, unless it refers to the new version of the merged casacore. He mentioned he had seen messages regarding changes in the format and/or implementation of casacore header files. I said I had no knowledge of that.
He had questions regarding imstat and max pos only being for entire cube (not per plane), I said probably not a difficult code change, but would likely break backward compatibility which is especially an issue because that change would be made in casacore. Pointed him to Jeff for further discussion.
He had viewer questions relating to annotations, I pointed him to Darrell and Susan.
He had questions about the best way to get lots of documentation reformatting, spelling, etc requests to the team. I pointed him to Mark.
We discussed our JIRA interactions and I suggested he update his email there since he doesn't regularly read the email to which JIRA sends notifications. I pointed out that it would benefit him and his project to respond in a more timely manner, since the faster he responds to my JIRA comments, the faster I can make further modifications he and his team need (it typically has been taking weeks for him to respond to my JIRA messages, and when I get input from him via JIRA, I typically respond to it within a day or two, which includes any necessary code modifications which he needs me to make). I think he has adopted my suggestion, since later in the week he was responding within 24 hours of me making a comment.
(CAS-7636) listobs failure on VLA ephemeris table: Measures issue. Passed to Jeff for reassignment.
Spent a couple of hours trying to understand why uvfits test which was present wasn't recognized by unittest system. Ville pointed out it wasn't in CMakeLists.txt. My opinion is that our convoluted build system obfuscates things like this.
(CAS-4451) exportuvfits garbles antenna positions: This is an issue in the ASTRON specialization of uvfits. AFAIK, I am not responsible for understanding or modifying ASTRON telescope (WSRT in this case) specific code. I therefore reassigned to Ger in the hope that he might know who is responsible for that code.
(CAS-1608) importuvfits does not load data with muliple antenna tables properly: Asked reporter to provide a uvfits file for which this behavior is observed.
(CAS-1844) exportuvfits gets frequency labels (headers) wrong if SpW are not in frequency order: Asked reporter to re-provide data which illustrates the behavior, since in the intervening several years, it has disappeared.
(CAS-2570) exportuvfits confuses velocities with multiple spws: Asked reporter for more specific location of data for which behavior is observed.
(CAS-7584) no random number seed in ia.addnoise() ?: Communicated to reporter that he will be unable to test this until after the casacore freeze has been lifted so that I can commit the code.
(CAS-7650) reimplement imval: opened ticket to reimplement imval, which is a huge horrible mess and so it is significantly higher cost to continually fix bugs reported against it rather than to simply reimplement it from scratch using reasonable software development practices.
(CAS-1376) Encode intents in UVFITS file (for AIPS): Got translation code from Bill Cotton. There are still issues with current requirements vs what the implementation currently does, so I reassigned to reporter for further requirements clarification.
(CAS-7648) imval(box='1,1,1,1') dies on 3D cubes that have no STOKES axis: This ticket was the impetus for CAS-7650 (reimplement imval). Provided user with workaround that met his requirements since I assert that fixing the current imval implementation is not a reasonable solution to this problem.
(CAS-4942) Time range in OBSERVATION subtable is zero length: Passed to Juergen for requirements gathering
Attended HPC mtg
Various spin-off meetings with visitors: weights and SD, VI2, statwt/uvcontsub in TransformingVi
set up work environment for casacore with git using Jim's description
found that it works
however, the github casacore and our "code", "asap", and "gcwrap" are out of sync. casacore from github doesn't link with them due to inconsistencies w.r.t. VI/VB. Notified Jim.
so casacore development is still halted for me
worked on making bnmin1 (part of wvrgcal) behave the same way on RHEL and OSX (CAS-7570):
after analysing the problems further, it occurred to me that the relevant code in Boost is a small subsection and is only contained in header files (inlined methods)
decided to try and simply incorporate the relevant code from Boost under a different namespace into bnmin1
Success! By only adding a few small header files to bnmin1 it was possible to make the RHEL and the OSX version produce identical random numbers and hence identical output.
Planning to use this solution for CASA 4.5. When time is found to remove Boost completely, this can be revisited.
Tested the spectral fitting in the viewer with an optical scientist and found a number of bugs. Filed umbrella ticket CAS-7676
Looked into CAS-7686
Debugged parallel cube tclean and got it working (tested only simplest case).
Kept trying to check in one last dataset into the data/regressions/ repo. Still trying.
Can't enable automated tests until this happens
Several minor bug fixes in tclean usage/interface + other non-casa work.
looked at file access pattern on MS written with MultiFile settings, columns end up randomly scattered in file in 4MB blocks which might be causing more seeks than normal and could explains the 5-30% slowdown seen when using multfiles.