-- DanielKlopp - 2012-10-01


This is a work in progress, please come back closer to October 8th, 2012 for a more complete guide. Everything written here until then is temporal and possibly untested. For most of the documentation we assume you are using a 3ware Inc 9650SE SATA-II RAID PCIe card. The instructions should be easy enough to follow with other card versions provided you update a few of the paths and numbers.

Installing 3Ware

Get the latest codeset from LSI. I tend to procure them from: http://kb.lsi.com/DownloadsCategory349.aspx

Download the codeset you require. I downloaded iso 9.5.4 for this example.

On your host (here naasc-oss-4.cv.nrao.edu):

mkdir /mnt/cdrom
mount -o loop 9.5.4-Codeset-Complete.iso /mnt/cdrom/
cd /mnt/cdrom/packages/installers/tools/linux/
# Handle user input as required below
./install --install
cd /mnt/cdrom/fw
tw_cli update fw=prom0006.img

Reboot the box. When it is back up, create the RAID arrays. I tend to do this from the command line as it is fast. The generic command line is tw_cli maint createunit. For naasc-oss-4, here were the commands with explanations following:

tw_cli maint createunit c6 rraid6 p0-5 k256
tw_cli maint createunit c7 rraid6 p0-5 k256
tw_cli maint createunit c8 rraid6 p0-5 k256
tw_cli maint createunit c9 rraid6 p0-5 k256 

That meant create RAID6 arrays on the 4 RAID controllers (c6, c7, c8, c9, the exact naming depends on the PCI layout and may vary from box to box). The option 'p' selects the physical disks of the controller (c6, c7, c8, c9) mentioned earlier. After that the option 'k' sets the stripe size. With RAID6 this means the RAID array has a 1 MB stripe width.

Stripe width and stripe size / chunk size are related but distinct terms. The stripe width is the number of disks times the stripe size. This is important as it relates to the blocksize. In short, you will want to set the filesytem creation options of stride width, stripe size and possibly offset. That, however, is way beyond the purpose of this page. For more details see Aurimas's blog on the subject.

Next, log in to the web interface. If firefox, IE, Chrome or any other web browser that has been updated in the past few years is used, then the browser will complain about a connection reset. This is referenced at http://kb.lsi.com/KnowledgebaseArticle16641.aspx . You can use an older version of the browsers, which is how I access the web interface for now. Type in the connection information for the browser, which in this example would be: https://naasc-oss-4.cv.nrao.edu:888 . Select the 'administrator' user and type in the default password of '3ware'. I would strongly advise changing this as the default setup.

Once in, select "3DM2 Settings" from the menu bar. Check on "Enable Email Alerts" and fill out the notification after the page reloads:

Sender: root@naasc-oss-4.cv.nrao.edu Recipient: $RECIPIENT_EMAIL SMTP Server: $YOUR_SMTP_SERVER

Change the two variables mentioned above as appropriate for your site. Click on "Save Email Settings" then click on "Send Test Message"

Then update the password to something other than the default "3ware" password.

Select MANAGEMENT->Controllers. Then use the drop down on the right hand side to select each controller and ensure the following settings are set for all controllers:

StorSav of Performance
Fastest I/O

Storsav descriptions can be found at: http://titan.physx.u-szeged.hu/opt/3ware/3DM2/help/en/Config_Units.6.14.html . Generically, I would use performance as any data center should have a battery backup system in place, and a proper shutdown procedure alleviating data corruption during a power outage.

The options of Fastest I/O and Adaptive depend upon your installation. We chose Fastest I/O because we make use of RAID6 and our disk replacement times are less than a business day.

3Ware Commandline

tw_cli, though I am thankful it exists, seems to have been developed at several different iterations. And at each iteration, the designers decided on a new syntax for the controller.

Upgrading firmware

For each controller on the box you will have to use the 'update fw' option to tw_cli with the path to the image file. For example:

[root@naasc-oss-5 ~]# tw_cli info

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
c4    9650SE-8LPML 8         6        1       0       1       1      -        
c5    9650SE-8LPML 8         6        1       1       1       1      -        
c6    9650SE-8LPML 8         6        1       0       1       1      -        
c7    9650SE-8LPML 8         6        1       0       1       1      -        

[root@naasc-oss-5 ~]# tw_cli /c4 update fw=prom0006.img

... text deleted ...

[root@naasc-oss-5 ~]# tw_cli /c5 update fw=prom0006.img

... text deleted ...

[root@naasc-oss-5 ~]# tw_cli /c6 update fw=prom0006.img

... text deleted ...

[root@naasc-oss-5 ~]# tw_cli /c7 update fw=prom0006.img

... text deleted ...

[root@naasc-oss-5 ~]#

The deleted text will show you the current firmware version, the version you are upgrading to, and whether or not the controller believes it is the correct version of the firmware.

You can run all updates at the same time, just start a new shell for each one.

Controller Versions

You can check the firmware version, bios version, serial numbers, and so forth of the controller with:

[root@testhost ~]# tw_cli /c4 show all
/c4 Driver Version =
/c4 Firmware Version = FE9X
/c4 Bios Version = BE9X
/c4 Boot Loader Version = BL9X


I've omitted some text partly because of the content, and partly because of the length of the output.

It is important to note that this command is run per controller. So the output will change depending on the controller being used.

Setting autoverify

One extraordinarily annoying oversight of 3ware (and I say this after fixing this oversight no less than 100 times) are lack of spare unit autoverify upon creation. After creating a spare unit, the unit is not automatically made with autoverify turned on. This is even after explicitly setting the autoverify on device creation via:

tw_cli maint createunit c4 rspare p5 autoverify

After creation you need to manually activate autoverify

tw_cli /c4/u1 set autoverify=on 
Topic revision: r2 - 2012-10-10, DanielKlopp
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback