Often in this space and elsewhere I’ve talked about the importance of measuring the effect of changes which you make to your website and product, and interatively changing based on the measurements. This is a critical skill. One thing that is even more critical is that you understand what the data would have looked like had you not made the change, because otherwise you could be changing for a spurious reason. As an example, I’m going to show you a real graph of my organic Google hits for 3 months:
You might conclude from this that my performance on Google is wildly erratic or that Google’s recently announced algorithm change in January has helped me greatly. Bzz, wrong answer. What is actually happening here is that you are seeing the combination of natural cycles in my market (low traffic over the Christmas break, low traffic on MLK Day and other long weekends, low traffic on every weekend) combined with a sustained massive sustained increase in overall traffic since early December. The cause of that increase is almost certainly related to me ranking higher for my core queries and getting on the first page for some very high traffic peripheral queries, such as bingo cards.
When you are making inferences based on the data for your own uISV , its easy to get caught in the trap of saying that any momentary spike is either a reflection of your own action or of some outside force (“The Google dance” is a favorite target for suspicion, as Google’s search results are capricious, poorly understood, and could potentially wreck you if you got on the wrong side of them). Resist the temptation — first, understand what the underlying pattern of your data (hits, conversions, sales, whatever) should be absent any influence from you or the suspected outside source. THEN compare what you are seeing to what you expected. For example, within the last week I changed the meta description on my index page to read more like my ad copy. I expect that this will eventually cause my click through rate in the organic search results to increase, resulting in more organic visitors. As of yet, focusing on just the last week of results, I do not have sufficient data to conclude that this change has produced a meaningful push in either direction.
