Gimpster & His Boring Blog

Making all other blogs seem exciting!
RSS icon Email icon Home icon
  • VMware vSphere ADAM/LDS issues after an in-place OS upgrade.

    Posted on July 26th, 2012 ashinn 1 comment

    I’m not a huge fan of doing in-place OS upgrades, but sometimes its just a necessary evil. Today I upgraded one of our vCenter servers from Server 2003 R2 64-bit to Server 2008 R2, which is a supported upgrade path from Microsoft.

    The OS upgrade itself went smoothly, but about 10 minutes after the final reboot vSphere Service Status alerted on these two issues:

    • LDAP replication health monitor – Failed to initialize LDAP instance manager
    • LDAP backup task monitor – JoinTool initialization error

    If you didn’t already know, vCenter relies upon Microsoft ADAM which was renamed Lightweight Directory Service. vCenter uses this as a repository for things like roles, license keys and many other metadata-ish stuff.

    I searched around the VMware KB & Google and really didn’t find anything useful. Then I dug through the vCenter Webservices Logs (vws.log), and ran into this:

    Action: Local ldap environment verification
    Problem: LDAP tools not found in “C:WindowsADAM”

    I zeroed in on: Problem: LDAP tools not found in “C:WindowsADAM and compared that directory between a known working vCenter on Server 2008 R2 and this problem child server.

    Sure enough 3 files were missing:

    • DSACLS.EXE
    • LDIFDE.EXE
    • REPADMIN.EXE

    After synchronizing that directory with the working server, I restarted vCenter Web Services and the errors went away! My assumption would be that during the upgrade those files were nuked as the LDS role was re-applied.

    For folks who might not know, here’s a VMware KB article for log file locations: KB1021804

    Till next time…

  • Check your UCS CPU heatsinks!

    Posted on May 16th, 2012 ashinn No comments

    Tonight I unboxed and installed several new UCS blades. I’ve done this quite a few times, and I’ve never had an issue. After inserting the blades into the chassis I went back to my desk to verify blade discovery. All was well, except one of the blades reported a “thermal-problem”. I removed the blade and its cover, and to my surprise the #2 CPU heatsink wasn’t secured from the factory!

    I bolted the heatsink down and reinserted the blade, but alas the error eventually came back. I’d say the CPU got fried when the FI’s turned on the blade to execute discovery. Oh well, back to Cisco it goes for repair.

    I’ll be checking the CPU heatsink status on every new blade I install from here out! Guess this B200 was built on a Friday.

    Till next time…

  • Resetting the ABS light on Nissan vehicles

    Posted on November 23rd, 2010 ashinn 1 comment

    Just making a note for future needs. My understanding is this works on pretty much any “modern” Nissan vehicle. It worked on my 2008 Nissan Frontier for sure.

    If you have the airbag light, it will show steady for a few seconds (the manual says 7 secs) after you turn the key from off to on (but not start) and then start to flash. Here is how to reset it:

    1. Turn the key from off to on.

    2. Watch the airbag light closely. It will stay on for a few secs. and then as soon as it blinks off (the start of the flashing), turn the key off instantly.

    3. Count to 5 (1 Mississippi, 2 Mississippi . . . .)

    4. Repeat steps 1-3 two more times until you have done the cycle three times.

    5. Now turn the key on and watch the light. It will start blinking in a different, slower manner. This is diagnostic mode. You can watch this for a little while, no hurry.

    6. Now turn the key off again, Count to 5 like in step 3 again, and turn back on. If the procedure worked, the airbag light will not be flashing.

    Thanks to msubullyfan on Nissan Forums for this!

  • FLARE 30 is out in the wild!

    Posted on September 3rd, 2010 ashinn No comments

    If you didn’t already know, after a few delays, FLARE 30 is now available for download on PowerLink. I’ve been playing with Unisphere a bit, and it seems like an improvement over ole Navisphere. However, it’ll take awhile to shed those old Navi ways. At the very least its a solid base to start fresh on.

    If I have any disappointment…. it would be how much DIDN’T change in Unisphere. In particular I was disappointed to see very little was done to improve analyzer. Lets hope thats something they build upon in later versions. Perhaps I was just expecting too much for a v1.0 release. Don’t get me wrong, I STRONGLY SUPPORT EMC revamping the midrange administraton tools.

    One small thing is Unisphere finally shows you what failover mode a host is at in connection manager. I realize it was no big deal to find out, but its one less proceedure to do. They also put the name of the mode along with the number (IE 4 = ALUA and so on).

    Also the way snapshots and clones are presented/organized is a pretty big step forward. I could go on and on, just download Unisphere and play around.

    I can’t tell you how it looks with the Celerra yet since none of my Celerra’s are on DART 6. If you try to administer a DART 5.x box, it launches a browser window to the control station (how nice of it!).

    I’m currently trying to figure out if a Celerra running DART 5.6.49 will work with FLARE 30 or not. I have a NS-960 basically not doing anything (who doesn’t?!), but I don’t want to brick the DART installation if I can avoid it! If/when I can get a straight answer out of EMC I’ll update this post.

    I really want to put FAST Cache and all VAAI through the paces before unleashing this day 1 code onto my poor customers & users. Hopefully I’ll get a thumbs up/down by this weekend from support and I can start doing some IO testing and report back.

    Of note I was told by an EMC TC that FLARE 30 = RecoverPoint upgrade, but I’ve yet to find a document to back that up. Just incase I bounced this CX/NS-960 out of the available RP splitters for now.

  • EMC CLARiiON CX Disk Offset Configuration

    Posted on March 17th, 2010 ashinn No comments

    I’ll be updating this with various OS method of setting the disk offset. This is mostly for me to consolidate my notes. It should be noted this is valid for MOST current EMC disk technologies, but you should always consult the documentation to make sure.

    If anyone has an OS to add, or sees an error let me know.

    Microsoft Windows Server 2003: 

    1. Start -> Run -> cmd.exe
    2. diskpart.exe
    3. List disk and find the new LUN you’re wanting to offset by number
    4. select disk #, where # = the LUN you wanted in step 3.
    5. create partition primary align=X, where X = 32, 64 or 128 (in my case, 64).
    6. format the disk in disk manager/assign a letter/use a mount point.

    Microsoft Windows Server 2008:

    Technically this is no longer required. This is because Server 2008 automatically sets the offset to 1MB on partition creation.

    Linux / older (2.x) ESX / etc:

    1. On service console, execute “fdisk /dev/sdX” (or “fdisk /dev/emcpowerX” for clariion systems), where X is the device on which you would like to create the new partition (a, b, c, etc).
    2. Type “n” to create a new partition
    3. Type “p” to create a primary partition
    4. Type “1” to create partition #1
    5. Select the defaults to use the full disk.
    6. Type “t” to change partition type
    7. Type “1” to select partition #1
    8. Depending on your Linux environment and need: type “83” to set type to Linux partition, or type “82” to set type to Linux swap, or type “8e” to set type to Linux LVM, or type “fb” to set type to VMFS (vmware file system). For other partition types, type “L” to display the list of codes.
    9. Type “x” to get into the expert mode
    10. Type “b” to specify the starting block of partitions
    11. Type “1” to select partition #1
    12. Type “128” to make partition to align on 64KB boundary (block No. 128)
    13. Type “w” to write new partition information to disk.
    14. Exit fdisk and format the partition with your favorie filesystem.

    Solaris:

    To be added.

  • Making Exchange 2007 perform on ESX.

    Posted on March 11th, 2010 ashinn No comments

    For a couple years we’ve maintained a full lab environment of our production hosted Microsoft Exchange 2007 CCR cluster. I have to be honest, and the Exchange administrator would agree, its never really performed that well. Finally the Exchange admin pretty much got ticked at the performance the other day and while he was out on vacation I thought I’d see what I could do. Its now performing about 100x the speed it ever did, and when he comes back Monday I hope he’s happy. More than anything we’ve just not had the time to really dig into the issue(s).

    Needless to say over these couple years we’ve all learned quite a bit about how to eek more performance out of ESX, and in particular Exchange on ESX. I thought I’d share a bunch of the concepts and tidbits in one spot I used to arrive at better performance. The old tricks of throwing RAM and vCPU’s at the problem just didn’t cut it.

    Its worth noting that people these days might not choose to use CCR on a virtualized environment (or never did), however I feel these concepts bleed over into stand-alone or maybe even FT/vLockStep implementations going forward.

    First and foremost, I invite you to read this article on Exchange 2007’s memory managment strategy:

    http://msexchangeteam.com/archive/2008/08/06/449484.aspx

    Okay, now that you’ve read that lets continue. Suffice it to say Exchange literally grabs every piece of memory and page it can…. if you let it (which most people do for cache/performance reasons).

    As most of you reading know, ESX has quite a few tricks up its sleeve in the memory management department itself and I invite you to read about those concepts in the vSphere/ESX manuals. Obviously the VM’s have access to physical RAM, shares RAM pages when possible, begs/borrows/steals from other VM’s (balloon) and when necessary swaps to disk as a last resort. After careful examination of the performance logs of the Exchange VM’s, it became very obvious ESX was swapping.

    Now, we happen to have an entire lab cluster and nobody really cares about performance … well nobody except the persnickity Exchange admin anyway. After doing some research I came to the conclusion I didn’t want the Exchange servers to swap memory … period. I then set a reservation on the VM to the exact same size as the RAM I’d granted. In this case 3GB. This effectivly disables the vswap since the host has no choice but to ante up.  Doing  just this provided an incredible performance boost to the Exchange cluster, but I scratched a little deeper.

    Within the VM itself I observed that it was paging quite a bit, and as the above referenced article shows… it always will. So to extract maximum paging performance I decided to create a couple LUN’s and mapped them raw (RDM) to the Exchange servers. I then did some research and came to the conclusion 4kb was the optimal block size for a raw paging volume and if anyone has differing opinions on that PLEASE post them. I then created page files equal to granted memory +20MB. After doing all of that and rebooting, I could tell we’re really cooking with gas now.

    Going forward I would like to talk to the Exchange adminstrator about migrating his VMDK based message stores to RDM as well.

    So, in closing … hopefully some of these ideas will help you come up with your own Exchange performance issue resolution. I’m sure there are more tricks I need to find, but right now I’m pretty happy with the results.

    Till next time…

  • ESX, Linux LVM, file system expansion and you.

    Posted on December 9th, 2009 ashinn No comments

    I see quite a few methods being used out there for logicial volume expansion. Everything from what I’m about to post all the way down to folks using cloning tools like Ghost and Clonezilla. Heres how I do it, and it works good for me. Note that I typically do these offline, for no other reason than I’ve biffed enough servers in my life to not be in a rush. If you’re needing 5-9’s uptime, look for another blog. It seems more than anything I have to expand / and so that also can play into this.

    Likely a good idea to visit the LVM man pages and/or this site if your LVM is a bit weak: read this and maybe also this.

    1. Shutdown the VM.
    2. Extend the VMDK you’re working with. You have a couple choices here. You can either extend it from the service console like so: vmkfstools -X 5G /path/to/the/vmdk (would add 5GB to that VMDK) or you can also extend it in the Virtual Infrastructure Client. If you use the service console, look at the options for vmkfstools for the exact syntax for your case.
    3. I think it goes without saying you should have backups and/or snapshots of the VM.
    4. Boot the LiveCD for your particular flavor of Linux. There are LVM LiveCD’s out there too, and also I bet a Knoppix disc might even work if in a pinch. I happen to run SLES & openSUSE pretty much exclusivly and the LiveCD’s work pretty well.
    5. Use fdisk and partition your new found space properly. Make sure you change the type to LVM (8e). If you’re not savvy with fdisk, you should likely stop now.
    6. So lets say the new partition you created is /dev/sda5. You now need to create a new physical volume, use the command: pvcreate /dev/sda5 and with any luck it’ll complete.
    7. Now you need to extend the volume group to the physical volume you just added. Use the command: vgextend *volumenamehere* /dev/sda5. Use vgdisplay if you don’t know which volume group you need to extend off the top of your head.
    8. Finally, its time to extend the logical volume itself. Use this command: lvextend -L 5GB /your/logical/volume/here. Use lvdisplay if you don’t know which LV you want to extend by heart. In this example I added 5GB to the volume. If you wanted to consume all space on the VG, you’d use something like this: lvextend -l +100%FREE /your/logical/volume/here
    9. At this point, if everything worked, you can expand the file system. On the systems I administrate we typically use XFS these days. XFS requires the filesystem to be online to grow. So while in the LiveCD you can quickly mount say /dev/system/root to /mnt and then execute xfs_growfs /mnt. Research the proper way to grow whatever file system YOU happen to use.
    10. Reboot the VM and make sure all is well.

    Till next time…

  • Why did sysprep get so hard for Server 2008, Vista and Windows 7?

    Posted on October 28th, 2009 ashinn No comments

    I know I’m way behind the curve on this, but why did sysprep have to get so difficult on the newer server and workstation operating systems? For quick hit deployments, the 20 line .ini file worked just fine. XML, the wave of the future. I’m thinking about writing a quick webpage that cranks out the XML for you without having to crack open WIM’s and install an almost 3GB deployment toolkit.

    That’s really all I had to say after a long lapse in posting 🙂

    Till next time…

  • Evicting a broken cluster node via command line.

    Posted on August 6th, 2009 ashinn No comments

    Like I’ve said before, I mostly have this blog for me. I often forget how to do stuff due to getting old.

    Ran into a situation where a Microsoft Cluster node ate itself, and I wasn’t able to evict it with the GUI Cluster Administrator. This second proceedure at this link fixed it up: http://support.microsoft.com/kb/282227

    If you cannot start the Cluster service, or if you have trouble removing the node, you can manually unconfigure the Cluster service:

    1. Run the Cmd.exe program to open a command prompt.
    2. At the command prompt, type cluster node nodename /forcecleanup, and then press ENTER.
  • Quick & dirty PXE boot server using openSuSE.

    Posted on June 7th, 2009 ashinn 1 comment

    Lately I’ve been dinking around with Clonezilla and other free disk imaging systems (more on that later), and using CD’s or USB keys can get tiring very fast. Besides Solaris Jumpstart servers, I’d really never gone down this path before. Turns out its quite simple, and you should be able to get it going in 20-30 minutes tops!

    I’m going to assume the readers of this have some cursory knowledge here. It should be noted that I added a DNS & Squid server to the mix here only because my server build VLAN is isolated from the world. I wanted to be able to patch them current after the image was deployed. If you don’t have those requirements, don’t install them.

    Quick network overview: My PXE boot server VM has one foot in a network with internet access, and the other foot is in an isolated build network. From this isolated network I image, patch and then scan the new machines for vulnerabilities.

    1.) Install the operating system. For this example I used openSuSE 11.1 on a pretty low-rent VM. Think 256MB of RAM and 4GB of disk or there abouts.

    2.) After you have the operating system installed, add the following packages with yast:

    • bind
    • yast2-dns-server
    • atftp
    • syslinux
    • dhcp-server
    • yast2-dhcp-server
    • squid
    • yast2-squid

    Yast will then complain about needing a bunch of dependencies, so just let it rip.

    Okay, so now that you have all of that installed.

    3.) Configure the DNS server with the appropriate forwarders (if you want) and all that jazz. However you like just so that the server can resolve. If you want, setup a fake dns zone like serverbuild.local. You could even setup a reverse zone for your bogus privately IP’d build network if you’re feeling really froggy.

    4.) Configure & start the DHCP server with appropriate IP range, IP to bind to (you most likely don’t want to have DHCP server on your public side!) and dynamic DNS to the aforementioned configured DNS server, should you so choose. The last three lines are PXE boot/tftp boot specific. Here is my /etc/dhcpd.conf:

    option domain-name “serverbuild.local”;
    option domain-name-servers 10.0.0.1;
    default-lease-time 14400;
    ddns-update-style none;
    subnet 10.0.0.0 netmask 255.255.255.0 {
    range 10.0.0.10 10.0.0.240;
    default-lease-time 14400;
    max-lease-time 28800;
    server-name “10.0.0.1”;
    next-server 10.0.0.1;
    filename “/tftpboot/pxelinux.0”;
    }

    5.) Configure & start the TFTP server. Once again I’d restrict this to your private side, unless you really want a TFTP server open to lots of folks (you don’t). Here is my /etc/sysconfig/attftpd:

    ## Path:    Network/FTP/Atftpd
    ## Description: ATFTP Configuration
    ## Type:    string
    ## Default: “–daemon ”
    #
    # atftpd options
    #
    ATFTPD_OPTIONS=”–logfile /var/log/atftpd.log –daemon –user tftp -v”

    ## Type:    yesno
    ## Default: no
    #
    # Use inetd instead of daemon
    #
    ATFTPD_USE_INETD=”no”

    ## Type:    string
    ## Default: “/tftpboot”
    #
    #  TFTP directory must be a world readable/writable directory.
    #  By default /tftpboot is assumed.
    #
    ATFTPD_DIRECTORY=”/tftpboot”

    ## Type:    string
    ## Default: “”
    #
    #  Whitespace seperated list of IP addresses which ATFTPD binds to.
    #  One instance of the service is started on each IP address.
    #  By default atftpd will listen on all available IP addresses/interfaces.
    #
    ATFTPD_BIND_ADDRESSES=”10.0.0.1″

    6.) Hopefully I’m coherent enough on the following steps:

    If it doesn’t already exsist, make a /tftpboot directory and copy /usr/share/syslinux/pxelinux.0 to /tftpboot. While you’re at it also make a directory called /tftpboot/pxelinux.cfg.

    7.) Create a file with the following contents named /tftpboot/pxelinux.cfg/default:

    default harddisk

    # hard disk
    label harddisk
    localboot 0x80

    # clonezilla live
    label clonezilla
    kernel vmlinuz1
    append initrd=initrd1.img boot=live union=aufs noswap noprompt ip=frommedia fetch=tftp://SERVERIPHERE/filesystem.squashfs

    implicit 1
    gfxboot bootlogo
    display message
    prompt 1
    timeout 600

    Change the SERVERIPHERE to represent your TFTP servers private IP.

    8.) Obtain the latest Clonezilla live cd .zip file, or one you’ve created custom (more on that later!). Unzip the .zip file in a location of your chosing and then copy live/vmlinuz1, live/initrd1.img and filesystem.squashfs to /tftpboot.

    In case you’re not in the know, vmlinuz1 = the kernel, initrd1.img = initial RAM disk image and filesystem.squashfs = the file system with all of the fun clonezilla stuff in it. I’m over simplifying things here, but whatever.

    For more details, see this link: http://www.clonezilla.org/clonezilla-live/livepxe.php

    9.) Create the message file you’ll see when your machine PXE boots named /tftpboot/message. This is what mine contains:

    Clonezilla PXE Boot

    To start the imaging system enter ‘clonezilla’ and press enter.

    Available boot options:

    harddisk – Boot from Harddisk (this is default)
    clonezilla – CloneZille Live CD

    Assuming you have all of the services started, you should now see this menu on the server(s) you wish to image. Type clonezilla and in theory the Clonezilla build environment should begin to load. This all works very nicely on my IBM blade servers.

    While in this example I focused on Clonezilla, you could replace (or add) any linux kernel & initrd (plus any other associated files) your little heart desires in the /tftpboot/pxelinux.cfg/default. Like for example a RedHat server build environment or whatever you might need. I assume you could easily shove in a BartPE/WinPE image too if you wanted.

    I’ll run through this again later in the week to make sure I didn’t miss something huge.

    I didn’t really get into configuration of the Squid proxy I guess, but there are plenty of HOW-TO’s out there for that!

    Till next time…