September 3, 2010

Happy Birthday, Internet.

x
Bookmark

Birth of the Internet is today! It turns 41

Birth of the InternetIs the name Leonard Kleinrock familiar to you? How about Robert Kahn? Larry Roberts?

September 2, 2010 marks the 41st birthday of the network that would become the Internet. I know you are a big fan of the Internet and networks, just like I am, so it is fortuitous that the 41st anniversary of the first packetized data to cross the first router (Interface Message Processors, or IMPs) is in just a couple of days.

I know what you’re thinking: Is the first message on the first router really something that should be commemorated? I mean, we don’t celebrate the invention of, say, concrete (although there are a lot of sites on the Internet dedicated to the history of cement and concrete!).

The first communication between the IMP, and a computer (in this case called an SDS Sigma-7), happened in the Network Measurement Center at UCLA. The IMP was build by a company that still kind of exists called Bolt, Berenak and Newman (although, now it’s called BBN Technologies). BBN has long been involved in defense contracting work in the US, and was founded by a couple of MIT professors and a former student.

The first communication to take place between two IMPs with attached computers, happened on October 29, 1969, between UCLA and Stanford University.

The first birthday message?

‘Lo’.

‘Lo’, the first message sent between the IMPs at UCLA and Stanford, was the result of a system crash. According to sources, it was supposed to be the word ‘Login’, but the system crashed and it took an hour to bring the system back up.

It’s not really like the first message sent, ‘Lo’ is really on par with,’Mr.Watson—come here—I want to see you.’ Is it?

I personally think it’s a close second.

The telephone revolutionized communication in the 19th century. Before the telephone, you had to send letters or possibly a telegraph. Both had and still have disadvantages.

Letters took a long time to across the country or across oceans. A long, long time. Telegraphs required an expert at each end that could receive and transpose the message from code to words. Telegraphs tended to be expensive.  Telegraphs, and Morse code, were technological game-changers in their own right, since communication could be almost instant. But it’s not the same a hearing a voice over a machine and network spanning miles and miles.

The telephone, as early as the 1890s, made it possible to speak to other people over long distances. Witchcraft!

Back to the Birth of the Internet:

In the ensuing 41 years, this nascent ARPANET, which had four ‘nodes’ by the end of 1969, has changed everything.

At roughly the same time, throughout the 1970s and ‘80s, computers themselves went from being monolithic entities at research institutions for crunching numbers to being a hobby, a vocation and even a profession for so many people. PCs and Macs made computers accessible to anyone with a little money and some time to learn. People, even at the start of the home computing era, realized that being connected to others over a network is way more fun than being on your computer all by yourself.

Even before graphics and free dirty pictures.

People wanted, and still want, to be able to communicate and share information in real time.

Myth-busting Time!

One thing that I have heard countless times is that the Internet was started to address a certain defense-related problem: In case of a missile strike on the US, there needed to be a decentralized network so that we could, essentially, fire back. A centralized control facility would be the enemy’s main target. We needed to be able to strike back even if one facility was destroyed.

Guess what?

Not so.

The book I am reading on this at the moment, ‘On the Way to the Web’, says this was not really the case at all.  People ended up confusing the ARPANET with a project being worked on by the Rand Corporation at the time. The Rand Corporation project was a voice network project, wherein the network could survive and function if some of the links were taken out. It did involve packetizing messages, so maybe it was an early form of Voice-over-X technology (since IP was not yet completed). I started thinking about this more, and it makes sense: at that time, missile sites were manned by men, not machines, so a resilient voice connection was more important than a primitive form of Email. The men in the silos would want to speak to one another. (See ‘Wargames’ if you have not already).

The lesson learned from this little tidbit? Not everything the ARPA (now DARPA) did was related to making war.

But this ode to the Birth of the Internet is not just about appreciating the Internet. The Internet made all the protocols we use on a daily basis necessary, which in turn gave many of us opportunities.

I personally say a quiet thanks to the inventors and engineers that made it all possible. When I stumbled in to the IT field nearly 15 years ago, I had no idea what the creation of the Internet entailed. I just knew that being able to understand some stuff about networks made it possible for me to provide for my family doing work that was interesting and fun.  I started to be able to appreciate the intricacies of networks and networking. I began to appreciate the elegance of simplicity, and found that some of the greatest challenges are the times when I have to fix someone else’s network design. I also became an ardent defender of networks, especially robust ones, when the System guys would cast stones. The finger is always pointed at the network, and we always end up having to prove it’s not.

Remember that from its humble beginnings, the Internet has remained a place for learning and information sharing.

So, today on the birth of the internet, when you click over to this post, or to any other of nearly countless web pages, remember to marvel at the Internet and what it has done for so many of us.

All Hail The Mother of the Internet!

x
Bookmark

When I am not studying to get better at my job or to pass a certification test (or being a husband and dad), I enjoy reading. A lot, actually. A majority of the books I end up reading are related to history; not just American History either. I love to read about inventors and explorers, about mythology and sociology. One of my favorite books of all time, Cryptonomicon is like an ideal mix of things I love: History, spies, and technology, covered in a hard candy shell of weirdness. If you’ve not read it, I would highly recommend it.

Back to my point: I find it interesting to read about the people that invented a certain technology or protocol, especially the ones I rely on. I like reading about the inspired team that developed the ARPANet IMP (Internet Message Processor, the first router), and about Vint Cerf and Bob Kahn and Linus Torvalds. I think it’s really cool to hear or read firsthand what these folks had in mind when they invented Ethernet or Linux.
It’s interesting because I am floored by these creations, and could not imagine being a pioneer in the same manner that these folks are.

Networking is (and was) without a doubt a male-dominated area of expertise, but one woman in particular changed the face of networks indelibly and her name is inextricably linked to a technology that is near and dear to my heart: the Spanning Tree Algorithm. The computer genius responsible for loop-free layer 2 topologies is a woman named Radia Perlman. She is sometimes referred to as ‘The Mother of the Internet’. I guess if there is a Father of the Internet (or multiple Fathers), and a Father of the Web (Berners-Lee), then there has to be a Mother as well, right?

If so, it’s Dr. Perlman for sure.

I bring her up because in the course of my Internet readings today, I saw that Dr. Perlman is receiving the Association for Computing Machinery’s SIGCOMM Award for 2010, for her contributions to networking that ‘we all use and take for granted every day’. She is delivering the keynote speech at the event. If you happen to be in New Dehli at the end of this month, stop in…

Dr. Perlman has received a lot of these awards in the past, I am sure. She’s gotten multiple patents in areas of networking and network security. But I think that the ACM SIGCOMM announcement hits the nail on the head: her contributions to network bridging and routing are fundamental and lasting.

I first heard of Radia Perlman in the late-1990’s, when I was given a book to read by my mentor and boss at the time. This boss was a firm believer in learning the fundamentals of networking, and the book I was tasked with reading was actually a mandatory assignment. It was a performance review goal, so I had to do it. Making it mandatory made me gripe and complain a little, but once I started reading that book, I was very glad that I did.

The book in question? ‘Interconnection: Bridges and Routers’ (it has since been retitled to add ‘Switches’ in there after ‘Bridges’).

Dr. Perlman’s writing style is light and easy to read (especially when compared with some of the other textbook-style books on networking, like ANYTHING by Stallings. Yawn).
She starts out where pretty much every networking book does, with the OSI model, but the difference in ‘Interconnections’ is that she emphasizes that the OSI model is a framework and guideline, not a set of inflexible, carved-in-stone rules.

Since she is the inventor of STA, I always recommend reading her book as a primer for those that are studying vendor-specific implementations of STP. Reading straight from the source what she was thinking and what she was intending and why she made certain choices along the way in her original design is totally worthwhile. She even includes a corny poem about STA. The whole book is worth reading, and leaves little room for doubt about the breadth of the author's knowledge.

I continued researching today about Dr. Perlman, and found that there is a great article from the MIT Admissions Blog and Investor’s Business Daily from a few years ago about Perlman. She has a Bachelor’s and Master’s in Math and a Ph.D in Computer Science, all from MIT.

The article talks a bit about what kinds of struggles she faced as a woman in the early days of her career. People tended to ignore her contributions, in spite of the fact that she is a trained and educated mathematician; the article tells of an incident in which Perlman presents a solution to a routing issue, spending 30 minutes detailing her solution using an overhead projector, to be completely ignored by the meeting organizers. It is fortunate that she chose to continue on.

The article goes on to explain that Perlman took a job at DEC, and was given a task involving getting nodes on a network to communicate. Other engineers at DEC had been working on the problem for months, with little progress and success. Perlman provided a solution to the problem the next day. I am not 100% sure, since the article is short, but I believe it was the beginnings of STA that were the solution.

Dr. Perlman also mentions, in the article, that teaching others and writing are great ways to keep challenging oneself and learning. To quote,’You can never learn enough, she believes’. She is correct. She also says that one of the best ways to learn something that is really going to stick in your mind, is to teach it.

The next time you are getting ready to add a LAN switch to your network, or set up an etherchannel, or just about any other change, give a ‘Shout Out’ and say a 'Thank You' to Spanning Tree, and its creator, The Mother of the Internet, Dr. Perlman.

Cryptonomicon
Overall Rating:
 
Retail Price: $8.99
Amazon Price: $4.49

Networking Careers and Investing in Yourself

x
Bookmark

In my long and storied IT career (haha!), I have seen many rounds of Reductions in Force (RIFs) and layoffs. When a company sells a division, spins off a business, or shows less-than-positive revenue, we invariably see some people, deserving or not, get their walking papers. I have to be completely candid and say that it sucks when there are layoffs. In some cases, especially now (in the worst economic climate in 50 years), there is nothing you can do about losing your position. You may be the best of the best, but sometimes it is just your turn.

There are a lot of things you can do to make yourself valuable to your current employer and desirable as an employee should you happen to be laid-off. Fear and uncertainty can paralyze you and render you ineffective. Guess what? Being ineffective is really bad, and can end up making you a prime candidate for dismissal. Or, you can take that fear/uncertainty/doubt, turn it into motivation, and commit to constantly learning and improving.

A good friend and mentor of mine had an email signature that I probably scoffed at when I first saw it years ago. It read: ‘Success is a journey, not a destination’. Regardless of who said it, I think it is a really relevant bit of wisdom these days.

To succeed, I have to keep getting better; to get better, I have to keep learning.

But how, you ask, might you do that? Thanks for asking! Here are some things  I find important to combat uncertainty.

  • Set professional certification goals for yourself. Ever heard the term ‘paper CCNP’ or ‘paper MCSE’? Yeah, me too. Certifications never guarantee you a job, and if you are not really an expert at the subject matter and claim to be, you will be exposed in short order. On the other hand, if you are a hard worker, have good references AND relevant certifications, you just might get a foot in the door. If you are not worried about losing your job, I believe it’s still vitally important to have cert goals to work toward. Certifications are a benchmark of your ability to learn material and pass a test, but certifications also show that you are willing to make an effort on your own time to acquire new knowledge and skills, to make an investment in yourself. I also strongly believe that you have to set cert goals for yourself; if your boss is setting the goal for you to become CCNA-certified, that is external to you and I feel you are not as likely to follow through and succeed.
  • Learn a programming or scripting language. This is very specific, but I can’t stress enough how important it is to show your current or future employers that you are versatile and can use your knowledge to innovate and automate. Being an expert in Cisco IOS or Juniper JUNOS is really great, but knowing how to write a SHELL or PERL script to take care of some routine function shows you value efficiency and are more versatile than the guy that memorizes router commands. It has been my experience that knowing scripting and programming is a desirable skill and sought after.
  • Cross train. Anybody out there old enough to remember Bo Jackson? He was the spokes-athlete for Nike’s cross training shoes in the late 1980s. He was both a pro American Footballer and pro Baseball player. His versatility was lauded. You can follow his example in your job and work closely with the folks that you support. Do you work with Linux or Windows admins? Learn about operating systems and processes. Build your own servers (VMWare and other similar systems are great for this), learn to find your way around, read the MAN pages. This kind of cross training will not only help you with learning about Cisco IOS (guess what? It’s an OS!!), but also with troubleshooting and debugging problems. Be a specialist in networking, but learn enough about the other stuff to be valuable as well.
  • Learn the Theory. It’s a drag to read RFCs in most cases, but I recommend it. Learning the theory and principles behind what we do in production networks is one of the best ways to add value. If your employer changed tomorrow from Cisco to HP networking gear, would your learning curve be steep? If you have only concentrated on Cisco-specific commands, then yes, probably very. If you know the theory behind a protocol, you can more easily discern what commands, regardless of vendor, you need to use to achieve a certain result.
  • READ A LOT. Most people are not readers of books these days, and many argue that the material in books is more often than not outdated and irrelevant. In some cases, that is true. Some are timeless, ‘classics’ of networking if you will, and should be read and re-read. If books are not your thing, find and follow reputable blogs. Take their content and do your own research. [I have to include a shameless, unsolicited plug: Safari Books Online is such a great investment in learning. You have access to literally thousands of books and videos. Check it out, it’s worthwhile.]
  • Don’t Ignore Layer 8. Layer 8 you ask? The political layer usually, although in this case, I use Layer 8 to mean the Business Coping Skills Layer. Learn about project management and business processes related to your work. Project management is a really great skill, and in most cases, you can continue to be a technical contributor as well. Any situation in which you can lead others and add value to projects is a good idea.

So there is my take on making yourself relevant and valuable. It is by no means comprehensive, and if your employer is laying people off, it may not be possible, in spite of your efforts, to stay employed there.  Be prepared, be knowledgeable, work hard, and always have a plan for when the unpleasant happens.

Network Troubleshooting: Structured is Best

x
Bookmark

network troubleshooting

Troubleshooting is not really something that many of us can easily learn from books, classes, or simulations. We usually learn to troubleshoot on the job, under pressure, and with very little room for error. How did you learn to troubleshoot network problems?

In my case, I was lucky to have had several veteran Networkers as mentors. Each of these folks did things in slightly different ways, but the emphasis of each was on taking an issue a layer at a time using the OSI Model. So in other words, each of these engineers was using a methodology for troubleshooting.

So much has been written about troubleshooting methodologies, that it’s really difficult to write any more about it without being redundant or sounding like I’m stealing ideas. My point is to emphasize the importance of having a PLAN when you attack an issue.

Cisco has outlined a comprehensive, multi-step process for troubleshooting. Those of you with any kind of background in the physical sciences will recognize that this Troubleshooting Methodology pays homage to the Scientific Method. Why not? It works!

  • Step 1: Clearly define the issue. We all know this is key, since most of the information that you are going to receive about an issue is secondhand from users that are not network specialists. Get as much background as you can to try and define the scope of an outage or issue.
  • Step 2: Gather detailed information and analyze it. This is a continuation of Step 1, but also the step where you could call upon your logging server, your SNMP tools, change control system, or any other monitoring system to get additional information. You may also need to go visit the users reporting the problem (if possible). Ask a lot of questions and get facts. Eliminate possible causes of the issue through your analysis (e.g. don’t concentrate on switch ports if your ISP link on your router is down).
  • Step 3: Define your plan of attack. By this point, you should have an idea of what might be going on based on your research and ‘Triage’ of the situation. Make a plan, and stick to it. If you have the time, make a diagram of the network you are working on; visualizing the network can often be helpful. On my team, we always start at the whiteboard.
  • Step 4: Implement your fix. Make ONLY one change at a time, and make only changes that are minimally disruptive to users that may still be functioning. It’s really important to make a single change to the network.
  • Step 5: Observe the results of your change or fix. Observe and ensure that the change or fix you implemented did not cause further issues.
  • Step 6: If your fix worked, document your steps and share with your team. Sound the ‘all-clear’ to the users, and monitor the network for any unanticipated side effects of your fix.
  • Step 6a: If your fix did not work, back out the change, and start over.

This is a paraphrased version of the method I have read on Cisco’s CCO, as well as in the CCNP T-Shoot Exam study guides. It’s really important to formulate and follow a plan for one very obvious reason: You want to know what change fixed the issue so that you can prepare for and prevent future issues.

It is my opinion that some people will try and convince other, less experienced people, that troubleshooting is an art or some mystical ability, and not everyone can get good at it. I will concede that some people make it look like an art or like magic, but really, it’s a learnable skill. The problem is getting practice at it, and honing your skills using your methodology.

There are a number of different approaches to troubleshooting once your methodology framework is in place. I mentioned early on that I was taught to use the OSI model layers, and I always used a ‘Bottom-up’ approach. It worked for me, since I was very familiar with Physical and MAC layer issues. Some will use a ‘Top-down’ approach, starting at the application and working down the stack. As always, your mileage may vary. If something is not working for you, find another approach that does. But always have a plan.

As you get more experienced, and if you are working on a network long-term, that you have helped build and maintain, you will be able to hone in on problems more easily based on your experience and background. It is still vitally important to resist the ‘Spray and Pray’ method, also called ‘shoot-from-the-hip’ troubleshooting. Making changes without a reason and without analysis is a recipe for disaster. You may get lucky once, twice, fifty times using this method, but it will eventually cause you issues. In the midst of a severe network issue, you may get frustrated and want to start making changes without planning. Take a moment and stop yourself. Deep breaths!

Troubleshooting network issues can be the skill that sets you apart and makes you a better networker. I highly recommend reading the CCIE Certification forum on CCO for more troubleshooting tips. I also highly recommend making a lab environment where you can mock-up and simulate issues so that you have the time to really get to the bottom of an issue without risking production downtime.

Preparing for Major Network Outages: Hope for the Best, Plan for the Worst

x
Bookmark

Image Credit: Despair.com

It is an unfortunate fact of life for Network Engineers that sometimes, in spite of our best efforts, networks go down. For some of us, this may actually be a life-threatening emergency (hospital s or military networks). Outages may result in lost revenue and damaged reputations. You could even lose your job if you were the guy that caused the major, unplanned downtime.

That leads me to the question: What are you doing in your day-to-day work as a Network Engineer to prepare for a major network outage?

Specifically, what tools, procedures or processes are you using to ensure that you are able to respond to a major issue quickly and effectively? Not everything in networking is as fun and sexy as setting up new MPLS VPNs. Some things we have to do are pure drudgery, but can also help you respond effectively when it counts.

Here are my Top-5 Items.

1. Network Change Control: Have you ever been in a situation where a coworker plugs a device in to the network and the network suddenly is just gone? Ever been the victim of a coworker that plugs in a new switch using VTP in Server mode, and your VLANs disappear? Ever had that sinking panic feeling in your gut the moment after you hit enter on the wrong router console? Me too. I know that a lot of people would argue that formal network change control introduces a layer of bureaucracy into an environment, and that getting teams together to build consensus is nearly impossible. I agree, but I also have seen network change control prevent not only problems but also duplication of effort. Of course, I am not saying that there is never a time or place for renegade networking (‘cowboy networking’), but change control notification makes it a lot easier to pinpoint issues that may have arisen as the result of a change. When you are under the gun to fix a down network, knowing what changed can be hugely important.

2. Documentation: Everyone hates doing it, but everyone needs it. Documentation is such an important part of network engineering, and there is no excuse for not having current, accessible documentation and diagrams published and backed-up. Personal experience has shown me the importance of documentation. A number of years ago, I joined a new team of network engineers, and of course I asked for copies of network diagrams and procedures. The answer I got from each member of the team: they are all in Bob’s brain. You see, Bob really liked that fact that no one else was able to draw out the whole network without a huge amount of effort. He felt important. I asked him to draw it up on my whiteboard and I immediately set about diagramming it (we use Visio) and checking the accuracy of it. Having good, current, accessible documentation should be nothing short of a requirement.

3. Network Monitoring and Baselining: Do you know what ‘normal’ looks like on your network? Knowing what the network looks like day-to-day, month-to-month, and so on, can give you a great deal of insight into issues that may be developing. There are some good free and commercial products that can generate reports about your network devices and their environmentals, port utilization, and processor utilization. SNMP traps may work for you, as might a suite of tools like Nagios. Netflow collectors are another good tool. The problem, if you will, with any of these items is that you have to read/look at each item for them to be useful. It is not likely that you will see patterns developing in your network if you don’t look into alerts and traffic spikes. Use caution though, it’s easy to get overloaded, which is worse than having nothing.

4. Logging infrastructure: Sure, this item could have been housed under number 3 above, but I am a true devotee of syslog, and specifically Syslog-NG. The first thing I do when dealing with access issues on my various firewalled networks is tail the log file, using GREP to find what I am looking for. Using the logs local to each device is great too, but I find that a unified log file (all my devices come into a main file) can help find issues, especially ones that span multiple devices. A bit of a disclaimer here: I have a lot of PIX and ASA devices in my networks, so quick access to the log files is essential and frankly indispensable for me. Logging using Syslog-NG also allows you to alert on logs using SWATCH or one of the other applications out there in Open-Source Land. Both SWATCH and Syslog-NG can be set up to show you things like ports being disabled due to your port-security policy, BPDU Guard policy, and duplicate IP addresses. Syslog is both a reactive and proactive way to monitor your network, and insanely useful. Two Enthusiastic Thumbs Up!

5. Troubleshooting Methodology: When that call comes in, you know, ‘The Internet is DOWN, Fix ASAP,’ how do you react? Do you hope that answers will pop into your head? Do you go out for a smoke? No, unless you are new to networking. You follow your troubleshooting methodology. For many of us, myself included, the methodology was not something written down anywhere. We had to learn, try and fail and try again and fail again. Troubleshooting is a learned skill, in spite what some folks will tell you, and it takes hard practice. Follow a consistent troubleshooting game-plan, and make adjustments if something isn’t working for you; Just don’t go into a downed network situation and try to use the ‘Spray and Pray’ method. Changing things at random, or without a good reason, is a really good way to create more problems and longer downtimes.

So there you have some items that I feel are important. We all have our own unique networks, so your mileage may vary. My overarching point is a relatively simple one: Knowledge is Power. Know what is going on in your network, and have a plan for when the unimaginable (or at least a really big outage) happens.