Working with LabTech

We’re in the midst of implementing a new business tool/offering through LabTech. A few years back, I worked with a company that had purchase “SilverBack”, a solution for MSP’s (Managed Service Providers) that allowed management and monitoring of remote sites. My thinking says that, although “good”, the advances in frameworks and functions have taken us to a new level. LabTech delivers a solution that provides for an incredible level of capabilities. Developed by an MSP for MSP’s, it shows.

The claim, “anywhere in three clicks” just might be true. Drill down within such easy grasp to see a config or report? Quickly done. Want to defrag a drive? Quickly done. Three clicks (as long as a dbl-click is counted as one, otherwise it’s four!!!).

Integration, depth, and usability are starting to show themselves as we continue to move forward with the initial implementation. Reports that allowed us to quickly determine which hosts on a network had Adobe Acrobat vs Adobe Acrobat Pro were quickly had.

Other functions that work for the Technician are the Redirectors. With choices. RDP, VNC, http, and https are a few of the choices one has when needing to access or to see how the end user sees certain web content.There’s even a set up to your LogMeIn account.

Scripts that allow removal of unwanted and unneeded “toolbars” are also available. Why we are continually pummeled by the likes of Google, Microsoft, Java, and other concerns to install another piece of their junk amaze me. LabTech lets me remove them. Remotely. Automatically. Kind of like a “search and seizure” mission.

I look forward to writing more about LabTech in the future, both near and far. I like choices. I like possibilities. LabTech appears to offer both!

What’s running on your network?

Are you a network administrator? Odds are good that the answer can be yes. Even if the only network you “run” lives at your home, you may be (by default) a network administrator.

Now, back to the first question: What’s running on your network?

Do you know? My network has computers and printers, a router/firewall with wireless capabilities. And an AppleTV, a BlueRay DVD player, iPhones and iPads…and should I take the time to reconfigure it, we’ve got a “Mole”–a wireless capable, infrared and motion sensitive, web cam. That’s in the house. We could talk about the office, yet I’ll save that!

When we talk about computers, we should also talk “OS”, or operating system. We run Linux (Ubuntu 11.10), Mac OS X (Lion and Snow Leopard), and Windows (almost all of which are on virtual machines). And yes, I know what’s running on my network. Right down to the application level. Why? Because I run a SonicWALL TZ 210 Wireless with the 5.8.x firmware.

It used to be that to have a clue, one needed to do “captures” of network data with a tool like WireShark or Snort. Some tool that would allow you to gather huge amounts of data to sift through. And sift through… The difference? Now I can look at a visual graph:

SonicWALL Real-Time Monitor

Visual monitoring view

Which might return a comment, “So what? There are all kinds of networking tools that show graphs.” And most all graphical tools, especially tools that show current traffic, have a value when it comes to getting that visual clue of what’s happening _now_. Yet you want to see more. The App Flow Monitor link, on the left menu just above our currently selected tool, breaks things down in a table that shows us flow rates, bytes and packets, sessions, and the application in use.

SonicWALL App Flow Monitor

App Flow Monitor

We can now see the applications used on our network. As long as our SonicWALL has been properly configured, we can see how our network is being used. With the modular, bundled approach of a SonicWALL, our network can now be protected at a level far beyond the capabilities of even a year ago. Yes, we can have Gateway Antivirus with its “6982259 signatures available on the cloud AV Database” (as of 11/6/2011), Content Filtering Service, Intrusion Prevention Service, Anti-Spyware, and Enforce Client Anti-virus, yet we can now have application level control on our network.

App Control now opens for us the ability to control how network applications use our network. We can give high priority and bandwidth, or we can restrict bandwidth. Or we can block an application, such as Skype with its port-hopping abilities, or we can allow Facebook, yet block Facebook games. That’s an awful lot of power for a home firewall (OK, so I got carried away!) And it’s more than is generally needed in a home environment, yet what about in a business?

From small business to enterprise, SonicWALL has an appliance to fit that need. The SonicWALL Next-Gen appliances are powerful, feature rich, and cost effective. Businesses have many of the same types of equipment on their networks as you have at home. Protecting a businesses assets has grown far beyond the requisite locks, keys and alarms. The reach of a business can extend beyond borders at the click of a button in a web browser or the opening of an attachment in an email. The consequences of failing to protect the network and computers of a business can be very costly. You, as a network administrator, are responsible for protecting your business–even if it’s your personal business at home, be it online banking, shopping, or simply surfing the net.

What’s running on your network? Given the state of Internet traffic today, it may be much more than you expect. Or want. Get control.

cPanel Server Migrations

Over the last few weeks I’ve done 3 server migrations. The first was a one-to-one upgrade, moving from an older server to a new one. The last two have both been two-to-one migrations: taking 2 vps’s (4 total) and consolidating to 1 dedicated (2 total).

In the first migration, we off-loaded DNS to DNSMadeEasy for management. I’ve advocated DME for the last couple years as a way to offload services and tighten security on a web server. They’re very cost-effective, have a quality service and are up 100% of the time due to their architecture. The biggest challenge was our attempt to build an IPSEC tunnel with Racoon to a Linksys RV042 router, which failed. Although we could see Phase 1 complete, Phase 2 failed regardless of how basic we made our configuration. Should someone have the secrets for getting CentOS and IPSEC to talk to a Linksys RV042, please share it with the world…

My second migration had off-site DNS of a different variety–which was quite cool as I simply made requests to the service provider and they filled the order.

In the third migration, we hosted DNS on the 2 VPS’s. This was quite fun as cPanel allows for Clustering of name servers. I made full use of this by syncing DNS data between all three servers. For practice, I redirected all of the DNS for one of the VPS’s to the new servers’ IP’s, added the proper data to the /etc/named.conf file that told the name server it was SOA (Master) for those domains, and shortened TTL’s to make site transfer and their inherent downtime manageable and negligible. This was also the source of my biggest oops: I forgot to update named.conf with this “master” information for the second server, which caused about a 4 hour outage.

Those were the primary differences in the migrations.

Although these migrations all coincided in a short time frame, I got control of the servers at different times. The similarities are that all three servers are running CentOS (one on version 5.2, the other two on version 4.6) and the initial setup.

Initial setup:

  • Install and configure CSF
  • Change sshd listening port
  • Walk through cPanel/WHM setup and modify for specific needs
  • Add mod_sec, suhosin, clamav, and recompile apache2 with needed extensions and php version
  • Set mysql root password

Once satisified with base configuration, I started transferring sites. Here, one of my daily tools came in very handy: Roboform. Even if you don’t purchase the full version, its ability to fill in a form makes it very worthwhile.

To transfer sites, I used the following method:

  • Add SSH port of far end server to Outgoing TCP in CSF
  • Access sites cPanel as root user (so that drop down box appears); ignore warning
  • Go to Backups >> Download or Generate a Full Backup
  • Select Secure Copy (scp) from drop down menu
  • Set email address as appropriate
  • Fill in Remote Server (either IP address or FQDN)
  • Fill in Remote User (in my case I used ‘root’)
  • Provide Remote Password
  • Set Port (same as the sshd port of the far end server)
  • Set Remote Dir to /home/
  • Generate Backup

If using RoboForm, all of this can be captured into an Auto-fill function for re-use, which is very convenient if you’re transferring a number of sites. All told, I transferred over 90 sites and RoboForm saved me a ton of time and typing.

  • On the new server, in WHM >> Backup >> Restore a Full Backup/cpmove file, put the username in the proper box and “Restore”.

Fortunately, WHM/cPanel makes it easy to determine the username: it’s listed after the date_ and before the .tar.gz.

Do pay attention to the restore window in WHM as a site builds…occasionally there are problems and most of them show up as an error. It’s also a very good idea to check the site and make sure that “it’s there”. There may be other issues, such as old, out of date scripts being transferred to a server that’s running new services; these may not be found until a change is attempted.

For one site, running MediaWiki, I wanted to update the package to the latest and greatest before I transferred it. Unfortunately, this didn’t work because the old server wasn’t running the correct version of PHP. Which segues into another “time saving” technique: before updating or upgrading a site, use the same technique above except to keep the backup tarball local. Then if your upgrade fails, terminate the site and re-install from your current backup. Make sure that you’ve moved the tarball outside of the users’ home directory though…terminating will destroy EVERYTHING in that users home directory, including backup files.

If for some reason you need to reinstall a site and you can use the current backup, you’ll need to move it from the /home/cprestore directory back to /home for WHM to see it. Here again, if a site fails to restore properly and the user has been created, you’ll need to “terminate” that user before attempting a new restore–whether from a newly transferred backup or a copy already located on the server in one of the allowed restore paths (noted in WHM).

Finally, if you’re hosting DNS on your cPanel server and you’ve set it up for clustering, it’s best to disable synchronizations BEFORE removing a site from the old server. Otherwise you may need to add a new DNS zone on your new server as synchronizing services, which is a part of the “terminate” process, will wipe out that zone where it’s needed most: the new server!

A new upgrade challenge–Dell Inspiron 1525

My latest upgrade challenge, taking a Dell Inspiron 1525 built on Vista and moving to Windows XP Pro, definitely requires persistence. With the XPS line, using the Dell Operating System disk P/N JY011 worked as expected to get the base OS on the computer. Previous experience showed that earlier versions failed to work because they didn’t have the proper drivers for the SATA controller and/or disk. With the 1525, formatting the drive failed with an error suggesting that the disk was corrupt.

nLite to the rescue! nLite allows the tech to create Windows installer cd’s that can be customized in a number of ways, including slipstreaming service packs, drivers, and setting options during the install instead of during startup. Did I mention it’s freeware? And well crafted, full featured freeware at that.

After gathering the needed info from lspci (a Linux tool that returns hardware info), I visited Dell’s support site and also Marvell and Intel to get the needed drivers. I also visited Microsoft’s download center and pulled Service Pack 3 for Windows XP. nLite created an ISO that I put on a CD-RW (at the painfully slow 4x speed) that got beyond the previous error point and copied the installation files onto the hard drive. Theory has it that this laptop should boot into Windows XP Pro running SP3.

Rebooting happened at 36 minutes after the hour and Setup gave the ever favorite “approximately: 39 minutes” message. Within 5 minutes and a couple of action screens later, there were 25 minutes left…approximately. Windows time is a marvelous thing when installing on a modern PC. Time flies and just about 12 minutes after the first reboot, the second reboot occurred which took us into the Windows world.

Driver errors still existed, yet after working through the basics–audio, video, network, et al–I still had one error that plagued me. “SMB Bus Controller” showed an error. Dell’s site was of no help, nor were other searches. This laptop has a built-in flash card reader, which uses Ricoh drivers…maybe that’s it, I wondered to myself.

My use of lspci was a bit lazy this time. I didn’t bother to mount a usb thumb drive to write the output of this marvelous little program. And guessing wasn’t getting me anywhere, so reboot with SysRescue once again. Paying particular attention to the output of lspci I found what I was looking for: Intel. Bingo! Voila! Hoopty Doo!

My first chipset install of Intel drivers was the wrong revision. One more trip to Intel’s website; one more download, this time for the proper “Chipset Identification Tool”.

In the spirit of the old west, putting notches on my laptop case would be appropo. In the new west, the IT west of todays modern computer-generated, ones-and-zeroes mish-mash of data, a blog will suffice. Notch one up for the blogger in me. Another laptop rescued from the clutches of Vista…

The value of the right tool

Yesterday I was dealing with a computer that had contracted the flu. An
exploit was causing browser requests to go to strange and far off
places, landing the user at web sites other than requested. Dialing up
my favorite online scanner at F-Secure
during a remote support session found 4 viruses and 2 spyware hits. And
all of a sudden, the browser would close killing my session and my scan.

One of the techniques I’ve used in the past to get around these issues
is to stop the scan after finding the first couple exploits and dealing
with them, then running another scan. Tried it. Browser closed killing
my session. And my scan.

Ran it again. This time the computer crashed. Although it didn’t “blue
screen”, it did reboot.

Sometimes working an issue is best done on site. Fortunately this
customer is both in the same state and city as I am so I drove over to
see what was going on. Every on-line scan did pretty much the same
thing: browser died, and the McAfee quarantine and logs showed nothing.
Also ran “HighJack This!”, where all I saw were the pretty normal
items, only a couple of which I items that even barely raised an
eyebrow (not even something to Google!). Except something was there.

While doing what I call “Walking the tree” (where I go into each users
directory in Documents and Settings and perform a slash-and-burn policy
on every Temp and Temporary Internet Files/Content.IE5 directory), I
all of a sudden saw a spike in CPU process. The one Explorer process
was running a consistent 50% CPU. Combined with the 50% that F-Secure
was demanding, the machine slowed to a crawl. Seems to me I woke
something up!

So I rebooted. System comes up and Explorer.exe continues to want its
50%. Shut down to a dead stop and start up. Same thing: 50% to
Explorer. Something was using Explorer to do its dirty work.

Enter the “Right Tool”. I like to keep a copy of the Sysinternals
Suite
on the servers we manage–comes in quite handy at times. This
was one of them.

Using Process Explorer (procexp.exe), I looked at the Properties of the
hungry, wayward Explorer process. A check of the threads tab showed one
thread using almost 100% of the consistent 50%. Here I had two choices:
Kill (oh, yes, how I wanted to just kill it!) or Suspend. This thread
was labeled “wsil32.dll”, which, according to the files properties and
a number of web sites, is a legitimate Windows file. It’s also one that
has apparently been exploited as it shows up consistently in a number
of spyware web sites. It’s also not a core file, so why not suspend it
to see what happens? Immediate relief. CPU drops to a very acceptable
couple percent–especially since I didn’t have anything running except
Process Explorer (and other more normal processes of the OS).

Given that success, I killed it. Still all runs well. To be on the safe
side, I renamed the file and moved it out of the system32/appcert
directory (just in case I need to restore it). Also rebooted the
machine. All running well.

Back to F-Secure and another on-line scan. After finding the first two
“spyware” hits, I stopped and cleared them using the “Recommended
Actions”  (aside from a ubiquitous tracking cookie, the one item of
concern was reported as Monitor.Win32.PrcView.) No crash. At
this point I gave the user some instructions and started another
scan–which was reported back to me as clean.

Although I’ve used a number of Sysinternal tools in the past, which
included playing with procexp, I’ve never used it to solve this level
of a problem before. And I’d certainly not recommend just willie-nilly
whacking executables and dll’s that seem to be whacky. Every
step should be taken to do the research and take the caution
appropriate. Process Explorer, though, does exactly what it says:
allows a tech to explore what’s going on with the processes of your
computer. And as far as tools are concerned, it just moved that much
higher in my tool box, my arsenal of tech tools that allow me to do my
job so that users can get to Google instead of some pathetic redirected
site passed on by an exploited computer.

Thanks Mark Russinovich and gang! You’ve made my life easier, improved
my technical skills, and, most important of all, empowered me to give
my Customers something to smile about: fix a computer so that it does
as it should–quickly and efficiently.

Enjoy the day.