Clean appears to be unstable when the pixel is smaller than about 0.1 of the synthesized psf fwhm.
The following data has synthesized beam ~0.015arcsec, rms of the visibilities =0.004, rms of a "well" cleaned image ~1.5e-7 Jy/bm off-source.
I tried a series at fixed angular image size - cell size decreasing and imsize increasing accordingly. When cell<~beam/7, I can clean very deeply (arguably too deep, but that's not the point here - the point is that I should
be able to clean to a couple of sigma, and even overclean, without it going completely haywire in the space of a few iterations). All images are from -1e-4 to 8e-4, and all residuals are from -1e-5 to 1e-4
cell=0.002" = beam/7.5
threshold = .0003mJy
this goes down to an rms off-source of 1e-7, and continued cleaning, while maybe not optimal, doesn't make it go crazy.
niter=1 000 000:
cell=0.001: we can get down to a few e-6, which is reasonable, but pushing it a little goes foobar fast:
with cell=0.0005, we can barely get down to 1e-5 before things to nuts, and there's never a good image with the right amount of flux. in fact, the flux even after a few iterations is twice what it should be...