For quit some time I’ve been wanting to have my own lab setup to use for my CCIE studies. The problem of course is that I don’t want to spend $10,000.00 to buy all real equipment. One day I finally decided I was going to take the plunge and build my own lab setup like so many have done before me. I should know let it be know that this will be the FIRST time I have ever built a computer, much less a computer for such a purpose. I used this site for a reference. Also, I am not Linux certified and I am still very much tied down to MS Windows due to my WoW habit (I never could get it to work on Linux). But before you think me completely crazy, I should state that I have had several years experience as a programmer (web and apps), and several more years as a computer technician/network engineer, so I’m not completely clueless. I read post after post (like this one, and this one) trying to wrap my head around exactly what I needed and how much it was going to cost. The below equipmnet list is what I decided upon.
Equipment List:
| ITEM | PRICE | QTY | TOTAL |
| CISCO CATALYST 3550 24-PORT | $ 89.00 | 2 | $ 178.00 |
| Cisco Catalyst 3560 Switch 24FE PoE | $ 240.00 | 2 | $ 480.00 |
| Sun Microsystems 501-4366 Quad Fast Ethernet 10/100 PCI 4-Port | $ 41.00 | 4 | $ 164.00 |
| ARK 4U-500-CA Black 4U Rackmount Case 3 External 5.25″ Drive Bays | $ 84.99 | 1 | $ 84.99 |
| BIOSTAR BIB75-AHB LGA 1155 Intel B75 SATA 6Gb/s ATX Intel Motherboard | $ 154.99 | 1 | $ 154.99 |
| CORSAIR Builder Series CX500 V2 500W ATX12V Power Supply | $ 59.99 | 1 | $ 59.99 |
| Intel Core i5-2500K Sandy Bridge 3.3GHz LGA 1155 95W Quad-Core Processor | $ 219.99 | 1 | $ 219.99 |
| CORSAIR Vengeance 16GB (4 x 4GB) 240-Pin DDR3 SDRAM | $ 89.99 | 1 | $ 89.99 |
| SAMSUNG 1TB 7200 RPM 32MB Cache SATA 3.0Gb/s 3.5″ Internal Hard Drive | $ 89.99 | 1 | $ 89.99 |
| STARTECH Anti-Static Wrist Strap | Model: SWS100 | $ 11.99 | 1 | $ 11.99 |
| STARTECH PLATE9M16 9 Pin Serial Male to 10 Pin IDC Female | $ 3.99 | 2 | $ 7.98 |
| STARTECH 1 port PCI Express 10/100/1000 Gigabit NIC | $ 27.99 | 1 | $ 27.99 |
| APC Back-UPS 550 | $ 76.99 | 1 | $ 76.99 |
| TOTAL: | $ 1,646.89 |
Software:
- Linux Mint 12 64-bit (later changed to Linux Mint 13 32-bit PAE … see below Problem #1)
- GNS3
- Oracle VirtualBox
- Yaquake
- Serial2Net
- Wireshark
- Screen
I’m sure this may look like over kill to some, however I wanted my lab to do more than just run GNS3 connected to real switches. I wanted my lab to be capable of all future endevours I plan to persue (e.g. CEH, Microsoft certs, CCVP, and any thing else that may come to mind). What was I thinking? The 4 x Sun 501-4366 QFEs is because many of the current CCIE practice labs call for 9 routers (R1-6 and BB1-3) which means I would need 18 connections to the switches, not just 12. Why the Intel i5 Quad Core processor? I wanted something up-to-date, but not too cutting edge. Why this motherboard? I can take 4 x PCI cards, 2 x Gig NIC, 2 x Serial, and still has lots of room for expansion. The case is nothing pretty, but I didn’t want pretty, I wanted a tank, a rack mountable tank!
Special Note #1: The Sun 501-4366 QFEs. The cards are fairly big and long. It is compatible with a PCI slot however the contacts do extend beyond the PCI slot. As a matter of fact, on card rests on a jumper switch and another atop the CMOS battery on the MOBO (not to worry, everything works just fine! Just keed this in mind when searching for a MOBO and QFE cards (again, this was my first computer build so I didn’t see it coming).
Problem #1: The Sun 501-4366 QFEs and malformed packets. If you plan to put these cards in a 64-bit system with more than 2GB of RAM, I foresee much pain and misery in your future! Unfortunately I did not know this before I chose to put these cards in my system (again, this was my first computer build so I didn’t see it coming). Basically there is a known issue with the Oracle/Sun Happy Meal (HME) driver that Linux uses for this card. Sun 501-4366 QFE + 64-bit system + >2GB RAM = malformed packets = no IP connectivity. This is Bug 10790. It took me sometime and a lot of reading other peoples posts to understand what the problem was because I could see the neighboring devices via CDP when terminal’d in to the router or switch. I chose the Sun QFE NIC because it was the best deal I could find. The D-Link 580TX was selling for $125-$250 on eBay (and from china no less). Did I mention this is the first time I ever used eBay. Simply put, I wasn’t ready to buy from China when I could get something here in the US for a quarter of the price. Those D-Link 580TX’s were starting to look pretty good however, I wasn’t willing to go into the hole for 4 of them just yet.
Solution: The good news is that there is a solution! I will spare you the details of the 3 days of mind knumbing disappointment I went throught trying all the various fixes that were detailed on other posts (lets just say I went as far as rebuilding the kernel as one post suggested, and since I’m not a Linux guru, I had to research that too). The kernel rebuild went fine however, it did not solve my problem. The fix. Install Linux Mint 32-bit w/the PAE enabled kernel! No, it does not necessarily have to be Linux Mint, Ubuntu 32-bit should work as well since to my understanding Mint is based on Ubuntu. Other Linux OS may work as well, but I can only vouch for Mint because that is the one I used. Other posts and bug fix websites did mention using a 32-bit OS but none mentioned using Physical Address Extension (PAE) which enables you to utilized all of you RAM. I first only loaded the 32-bit regular kernel and was limited to 2GB of RAM. If that had worked, I may have stopped there and just accepted the loss of the other 14GB of RAM I had installed, but it did not work. All the cards and NIC ports were not recognized by the regular 32-bit kernel. I was just lucky enough to see that PAE was an option in the GRUB bootloader but I did not know what it was at the time. I googled PAE, decided to give it a try and TA DAAAHH! All the cards and NICs were recognized and I nolonger had malformed packets so IP connectivity worked like a charm. I also had full use of all 16GB of RAM so nothing went to waste!
Problem #2: The Sun 501-4366 QFE ports using the same MAC address and unconsistant naming scheme. This is a problem because the the use of the same MAC by all the ports on the QFEs causes MAC address errors on the switches. The inconsistant naming scheme for the NICs (e.g. rename12, rename17, and out of order eth4 numbering) makes it a pain to assign the right ports on GNS3 and could lead to longer troubleshooting in the future.
Solution: The steps below outline a process I use based on the info gathered from many posts like this one and this one. No one post had this process, but instead had only pieces of the process. At first glance it may appear that there are unnecessary steps involved however, I did two things during my build that could cause problems if these steps are not followed in their entirety. First I changed from Linux Mint 12 to Linux Mint 13. That may not seem like an issue, but it was because there was a slight change to the attributes in the UDev database that are used to identify the NICs in the /etc/udev/rules.d/70-persistent-net.rules file. Secondly, I later added a 1 port Gigabit PCI Express card to the system. This caused the PCI BUS IDs to shift. Long story short, if you follow these steps, you will be able to ID and change the MAC and name of the correct NIC everytime.
Step 1: Use command “lspci” to enumerate pci components with BUS_ID and drivers.
Example: $ lspci | grep Ethernet
00:19.0 Ethernet controller: Intel Corporation 82579V Gigabit Network Connection (rev 04)
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit
05:00.1 Ethernet controller: Oracle/SUN Happy Meal 10/100 Ethernet [hme] (rev 01)
05:01.1 Ethernet controller: Oracle/SUN Happy Meal 10/100 Ethernet [hme] (rev 01)
05:02.1 Ethernet controller: Oracle/SUN Happy Meal 10/100 Ethernet [hme] (rev 01)
<lines omitted for brevity>
Step 2: Use command “find /sys -name net” to get the complete PCI Bus ID for the corresponding ethernet controllers from the sysfs directory.
Example: $ find /sys -name net
/sys/devices/virtual/net
/sys/devices/pci0000:00/0000:00:19.0/net
/sys/devices/pci0000:00/0000:00:1c.2/0000:02:00.0/net
/sys/devices/pci0000:00/0000:00:1c.3/0000:03:00.0/net
/sys/devices/pci0000:00/0000:00:1e.0/0000:04:01.0/0000:05:00.1/net
/sys/devices/pci0000:00/0000:00:1e.0/0000:04:01.0/0000:05:01.1/net
/sys/devices/pci0000:00/0000:00:1e.0/0000:04:01.0/0000:05:02.1/net
/sys/devices/pci0000:00/0000:00:1e.0/0000:04:01.0/0000:05:03.1/net
/sys/devices/pci0000:00/0000:00:1e.0/0000:04:02.0/0000:06:00.1/net
<lines omitted for brevity>
Step 3: Export the UDEV database to a file.
Example: $ udevadm info –export-db > udevdb
4. Open the exported UDEV DB and find the NIC by searching for the BUS_ID within the export file.
Example: $ gedit udevdb
4-a. Using the NIC device_path found in the UDEV DB, do a query to find attributes to match-on in the 70-persistent-net.rules file.
Example: $ udevadm info –query=all –attribute-walk –path=<found in UDEV DB>
5. Modify the UDEV rules file to assign a specific NAME to each NIC port based on matching criteria from step (4-a) (e.g. BUS_ID..).
Example: sudo nano /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.
# Assign unique NAME to the QFE (Quad Fast Ethernet PCI NICs) and on-board interfaces
KERNELS==”0000:03:00.0″, SUBSYSTEM==”net”, NAME=”lan”
KERNELS==”0000:02:00.0″, SUBSYSTEM==”net”, NAME=”eth1″
KERNELS==”0000:00:19.0″, SUBSYSTEM==”net”, NAME=”eth2″
KERNELS==”0000:05:00.1″, SUBSYSTEM==”net”, NAME=”qfe-a0″
KERNELS==”0000:05:01.1″, SUBSYSTEM==”net”, NAME=”qfe-a1″
KERNELS==”0000:05:02.1″, SUBSYSTEM==”net”, NAME=”qfe-a2″
<lines omitted for brevity>
6. Assign the a MAC ADDRESS to each QFE interface in the “/etc/network/interfaces” file.
Example: $ sudo nano /etc/network/interfaces
auto lo eth1 eth2 qfe-a0 qfe-a1 qfe-a2 qfe-a3 qfe-b0 qfe-b1 qfe-b2 qfe-b3 qfe-c0 qfe-c1 qfe-c2 qfe-c3 qfe-d0 qfe-d1 qfe-d2 qfe-d3
iface lo inet loopback
# Set all NICs to MANUAL ip address assignment and specify MAC ADDRESS for QFE interfaces
iface eth1 inet manual
up ifconfig $IFACE hw ether 00:01:02:03:04:E1
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
iface eth2 inet manual
up ifconfig $IFACE hw ether 00:01:02:03:04:E2
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
iface qfe-a0 inet manual
up ifconfig $IFACE hw ether 00:01:02:03:04:A0
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
iface qfe-a1 inet manual
up ifconfig $IFACE hw ether 00:01:02:03:04:A1
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
iface qfe-a2 inet manual
up ifconfig $IFACE hw ether 00:01:02:03:04:A2
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
iface qfe-a3 inet manual
up ifconfig $IFACE hw ether 00:01:02:03:04:A3
up ifconfig $IFACE 0.0.0.0 up
up ip link set $IFACE promisc on
<lines omitted for brevity>
7. REBOOT
Other than the two huge issues mentioned above, everything else went fairly smoothly with my build. The only issue I am currently having at the time of this writing is with the 2 9 pin DB9 to 10 pin IDC MOBO serial headers. They just don’t work. I researched the MOBO settings and can find nothing wrong. The 2 built-in serial ports work perfectly. This is not a game-ender, I’ll probably just switch to 2 USB to Serial cables. Judging from some posts I’ve read, I’m not the only one having issues with MOBO 10 pin IDC serial ports.
I hope this post was helpful the you, chances are if you came to this page via google search, you are looking for solutions to similar issues. Even if you do not find the solution on this page, you should click on all the links in the post and read what others before you have done, that may provide the missing piece to your problem.
Tweqouu 