Evaluating LIO Linux iSCSI target

I'd like to share some notes on LIO, a new iSCSI target in Linux kernel, as there are not so much of information about it.

1. General information
2. Concepts
3. Using targetcli
4. Using rtslib
5. Resizing LUN
6. Protect against split-brain
7. Summary

General information

Before LIO, the iSCSI target implementation in Ubuntu was iET.  Compared to iET, LIO is implemented as a pure kernel driver.  Operations for LIO is done via configFS special filesystem.

These packages are related to LIO:

• targetcli
  provides targetcli command-line utility.  This is the standard way to manipulate LIO.
• python-rtslib
  provides rtslib, a full-fledged python API library over configFS.

Web sites:

• http://linux-iscsi.org/
  The project main web site.
• A tale of two iSCSI targets
  How LIO replaced other iSCSI implementations.

Concepts

In addition to the standard iSCSI concepts, you should know some LIO specific ones.

For the standard concepts, read RFC3720.

• Fabric
  LIO supports several other fabrics than iSCSI.
  Look for /var/target/fabric/ directory for available fabric specifications.
• Backstores
  LIO supports several storage types as backing storage for LUNs.
  Specifically, PSCSI passes SCSI commands through to a (real) SCSI device.
  IBLOCK emulates SCSI devices on top of block devices such as LVM logical volume.

Using targetcli

How to setup:

$ sudo apt-get install --no-install-recommends targetcli python-urwid
$ sudo reboot

To avoid a bug in targetcli package, python-urwid package need to be installed explicitly.

Basic usage is described in this page:

• http://linux-iscsi.org/wiki/Targetcli

There are few other documentations or manuals for it.

Use context help inside targetcli command as follows:

$ sudo targetcli
/> help
(snip)
AVAILABLE COMMANDS
  The following commands are available in the current path:

    - bookmarks action [bookmark]
    - cd [path]
    - exit
    - get [group] [parameter...]
    - help [topic]
    - ls [path] [depth]
    - pwd
    - refresh
    - saveconfig
    - set [group] [parameter=value...]
    - status
    - version

/> cd backstores/pscsi
/backstores/pscsi> help
(snip)
AVAILABLE COMMANDS
(snip)
    - create name dev
    - delete name

> help create
SYNTAX
  create name dev

DESCRIPTION
  Creates a PSCSI storage object, with supplied name and SCSI device. The SCSI
  device dev can either be a path name to the device, in which case it is
  recommended to use the /dev/disk/by-id hierarchy to have consistent naming
  should your physical SCSI system be modified, or an SCSI device ID in the
  H:C:T:L format, which is not recommended as SCSI IDs may vary in time.

Using rtslib

First, there is a bug in the current rtslib (2.1-2).  Fix it, or you will fail to set some parameters.  Since targetcli is built on top of rtslib, this bug affects targetcli too.  Apply the following patch:

--- /usr/lib/python2.7/dist-packages/rtslib/node.py     2012-06-06 17:59:41.515308657 +0000
+++ node.py     2012-06-06 17:59:50.185146504 +0000
@@ -189,7 +189,7 @@
                               % str(parameter))
         else:
             try:
-                fwrite(path, "%s \n" % str(value))
+                fwrite(path, "%s\n" % str(value))
             except IOError, msg:
                 msg = msg[1]
                 raise RTSLibError("Cannot set parameter %s: %s"

HTML API documentation is available at /usr/share/doc/python-rtslib/doc/html/.
Run python web server as follows; point your browser to port 8000 to read documents.

$ cd /usr/share/doc/python-rtslib/doc/html
$ python -m SimpleHTTPServer &

To initialize iSCSI fabric:

from rtslib.target import *
 
fabric = FabricModule('iscsi')
if not fabric.exists:
    for mod in fabric.load(): pass
if not fabric.exists:
    error('failed to load iSCSI fabric modules')

To create a storage object:

from rtslib.tcm import *
 
bs = IBlockBackstore(0)
storage = None
for s in bs.storage_objects:
    if s.name == 'NAME':
        storage = s
        break
if storage is None:
    storage = bs.storage_object('NAME', '/dev/ubuntu/VOLUME')

To create a target, target portal group, LUN, and network portal:

from rtslib.target import *
 
target = Target(fabric, 'iqn.2012-06.com.cybozu:NAME')
tpg = TPG(target, 1)
if len(tpg.luns) == 0:
    tpg.lun(0, storage, 'NAME at HOST')
tpg.network_portal('IP', 3260)
tpg.set_attribute('authentication', '0')
tpg.set_attribute('generate_node_acls', '1')
tpg.set_attribute('demo_mode_write_protect', '0')
tpg.set_parameter('InitialR2T', 'No')
 
tpg.enable = True

Resizing LUN

Unlike iET, LUN can be easiy extended by using lvresize. Initiators need to rescan the session, though.

INITIATOR$ sudo blockdev --getsize64 /dev/sdb
1073741824

TARGET$ sudo lvresize -L 2g /dev/ubuntu/test

INITIATOR$ sudo iscsiadm -m session -R
Rescanning session [sid: 1, target: iqn.2012-06.com.cybozu:test, portal: 10.xx.xx.xx,3260]

INITIATOR$ sudo blockdev --getsize64 /dev/sdb
2147483648

Protect against split-brain

In our environment, iSCSI initiators are virtual machines which, when malfunctioned, will be restored on another server.  That means two or more same VM instances can run at once.  It is disastrous that these VMs write to one iSCSI target LUN at the same time.

To eliminate such possible split-brain problems, we restrict a target portal group (TPG) can have just one session by applying this patch to LIO.

--- linux-3.2.0/drivers/target/iscsi/iscsi_target_login.c       2012-06-07 05:37:34.000000000 +0000
+++ iscsi_target_login.c        2012-06-07 05:34:22.866487169 +0000
@@ -1107,6 +1107,20 @@
                goto new_sess_out;
        }
 
+       /* Cybozu */
+       if (zero_tsih) {
+               int error = 0;
+               spin_lock_bh(&np->np_thread_lock);
+               if( tpg->nsessions > 0 )
+                       error = 1;
+               spin_unlock_bh(&np->np_thread_lock);
+
+               if( error ) {
+                       pr_err("Detected possible split brain\n");
+                       goto new_sess_out;
+               }
+       }
+
        if (zero_tsih) {
                if (iscsi_login_zero_tsih_s2(conn) < 0) {
                        iscsi_target_nego_release(login, conn);

This is quite a dirty and imprecise hack, but just enough for us.

Note that MaxConnections session parameter does not prevent this problem.  Even if two iSCSI initiators share the same IQN, sessions may be identified differently because iSCSI sessions are identified by a random numeric session ID.

Summary

This article describes LIO and related API and a utility.
Bugs lurking in the current Ubuntu are also addressed.

A kernel patch to restrict session numbers for a TPG is also presented.

Hope this helps.

Yamamoto, Hirotaka @ymmt2005