Tuesday, 12 June 2018

Kernel Hacking to achieve custom FC frames - incomplete

This post covers the incomplete work I did on trying to modify FC drivers to send custom FC frames. I hope someone can pick it up and use it because I don't think I have time to finish it.


To determine what symbols have been exported by the kernel, run:


library functions -> run in user space
system calls -> run in kernel mode

Connecting to the SAN:

[425303.444353] lpfc 0000:07:00.0: 0:1303 Link Up Event x1 received Data: x1 xf7 x10 x0 x0 x0 0
[425305.532359] scsi 6:0:0:1: Direct-Access     LIO-ORG  storage1         4.0  PQ: 0 ANSI: 5
[425305.532945] sd 6:0:0:1: Attached scsi generic sg1 type 0
[425305.534874] sd 6:0:0:1: [sdb] 1048576000 512-byte logical blocks: (537 GB/500 GiB)
[425305.534878] sd 6:0:0:1: [sdb] 4096-byte physical blocks
[425305.535976] sd 6:0:0:1: [sdb] Write Protect is off
[425305.535981] sd 6:0:0:1: [sdb] Mode Sense: 43 00 10 08
[425305.536109] sd 6:0:0:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[425305.605769]  sdb: sdb1
[425305.606899] sd 6:0:0:1: [sdb] Attached SCSI disk
[root@localhost new-modules]# ls -la /dev/sdb*
brw-rw----. 1 root disk 8, 16 May 12 16:10 /dev/sdb

brw-rw----. 1 root disk 8, 17 May 12 16:10 /dev/sdb1

From within: /usr/src/linux/Documentation/devices.txt
  8 block       SCSI disk devices (0-15)
                  0 = /dev/sda          First SCSI disk whole disk
                 16 = /dev/sdb          Second SCSI disk whole disk
                 32 = /dev/sdc          Third SCSI disk whole disk
                240 = /dev/sdp          Sixteenth SCSI disk whole disk

                Partitions are handled in the same way as for IDE
                disks (see major number 3) except that the limit on

                partitions is 15.

Finding the Link Up Event within the driver:

[root@localhost lpfc]# grep "Link Up Event" *
lpfc_hbadisc.c:                                 "1303 Link Up Event x%x received  

Function that brings the link up:
lpfc_mbx_cmpl_read_topology(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb)
        struct lpfc_vport *vport = pmb->vport;
        struct Scsi_Host  *shost = lpfc_shost_from_vport(vport);
        struct lpfc_mbx_read_top *la;
        MAILBOX_t *mb = &pmb->u.mb;
        struct lpfc_dmabuf *mp = (struct lpfc_dmabuf *) (pmb->context1);
                        lpfc_printf_log(phba, KERN_ERR, LOG_LINK_EVENT,
                                        "1303 Link Up Event x%x received "
                                        "Data: x%x x%x x%x x%x x%x x%x %d\n",
                                        la->eventTag, phba->fc_eventTag,
                                        bf_get(lpfc_mbx_read_top_link_spd, la),
                                        bf_get(lpfc_mbx_read_top_mm, la),
                                        bf_get(lpfc_mbx_read_top_fa, la),

Looking for file operations in the driver:

[root@localhost lpfc]# grep fops *

lpfc_init.c:    .fops = &lpfc_mgmt_fop,

vi lpfc_init.c
static const struct file_operations lpfc_mgmt_fop = {
        .owner = THIS_MODULE,

static struct miscdevice lpfc_mgmt_dev = {
        .minor = MISC_DYNAMIC_MINOR,
        .name = "lpfcmgmt",
        .fops = &lpfc_mgmt_fop,

[root@localhost lpfc]# cat /proc/kallsyms | grep lpfc_mgmt_fop

ffffffffa0162c60 r lpfc_mgmt_fop        [lpfc]

Setting up kernel source to modify drivers:

[kylie@localhost ~]$ uname -r

[kylie@localhost ~]$ koji download-build --arch=src kernel-4.4.8-300.fc23.x86_64
kernel-4.4.8-300.fc23.src.rpm                                                              | 168 MB  00:02:26 !!!

[kylie@localhost ~]$ ls
kernel  kernel-4.4.8-300.fc23.src.rpm  rpmbuild

[kylie@localhost ~]$ su -c 'dnf builddep kernel-4.4.8-300.fc23.src.rpm'



Different tact - using fcoe

1. Using fcoe tools to setup a ethernet interface with DCB to enable fcoe:

This example configures interface eth3 to automatically connect to storage over a discovered VLAN.

1) Configure FCoE on the interface
     # cd /etc/fcoe/
     # cp cfg-ethx cfg-eth3

2) Start lldpad and configure the interface for DCB.
    # service lldpad start
    # dcbtool sc eth3 dcb on
    # dcbtool sc eth3 pfc e:1
    # dcbtool sc eth3 app:fcoe e:1

As a convenience there is a script that will confirm if DCB has been configured correctly for FCoE. The script is run as follows,

    <fcoe-utils source>/debug/dcbcheck.sh eth3
    (note: this is on the root device, not the VLAN)

Follow the suggestions and repeatedly run the script until it states that DCB is configured correctly.

3) Start fcoe
    # service fcoe start
      After a few moments your storage should appear (assuming everything is
      configured correctly on the fabric)

4) Setup lldpad and fcoe to start when booting
     # chkconfig lldpad on
     # chkconfig fcoe on

2. Connect port to an FCoE switch set to span another port

need to source an FCoE switch... maybe on in datacentre? :/

3. Record the traffic

Use scapy to record a pcap - or record via wireshark

4. Replay traffic

Use scapy to replay traffic on the FCoE enable ethernet interface

5. Make changes to FC frames

No comments:

Post a Comment