An international consultant

Research methods for Information Technology

I spent Tuesday at Fondren Library, Rice University, researching research methods. Three years ago, a colleague and I wrote a book on logical data modeling (Logical Data Modeling: What It Is and How To Do It, Springer, 2005). A couple of weeks ago, we talked about writing one on research methods in IT, as a text for beginning graduate students. Hence the trip to the library.

My own notion was that the target audience was students looking toward a business career. Thus, the book should focus on common areas of business research:

• Technologies
• Vendors
• Products
• Standards
• Benchmarks
• Case studies
• Etc.

Researching such topics in a business context often means that it's got to be done quickly, cheaply, with the facilities at hand. You've got to be able to separate marketing hype from fact. You've got to handle how different vendors use different vocabularies to describe roughly the same thing. You've got to choose an appropriate way to present your research. Thus does research blend into technical/business writing.

But if the student is on the road to an academic career, the rules change. That was the message of the books I examined.

First: C. William Emory, Business Research Methods, rev. ed. (Irwin, 1980). This book is in its 4th edition, but not to worry. Its objectives are to

• Provide students and managers with conceptual tools and techniques (1) to understand how the scientific method is applied to business issues and (2) to evaluate research proposals and studies from a design and execution perspective
• Provide professors with a basic text in research methods, one that will provide sufficient depth and breadth of coverage to free them from methodological detail. [In other words, to be a reference as well as a teaching text.]

The book is divided into four sections:

1. Foundations of scientific thinking, nature of business research, notions of concept, hypothesis, theory, and reasoning process
2. Research design: process, measurement, development of indices, sampling design
3. Data collection procedures: library searching, questionnaire development, scaling, observation research, experimental design
4. Data analysis and reporting of research.

As would soon become apparent, Prof. Emory's approach is the dominant one in academic circles: "positivist" or "analytic." Truth is the assembly and interpretation of relevant fact. The purpose of research is to contribute to the body of knowledge, and that can be done only by adhering rigorously to a standard identified with "scientific method."

The difference between academic research and practitioner research couldn't be more obvious. I'm reminded of a paper by Herbert Simon that poses the question "If your research shows that A is better than B, but the difference between A and B is not statistically significant, what do you do?" The usual answer: Increase sample size. His answer: Choose A. You may be wrong, but in a series of such choices, you'll be right more often than wrong, and you will have saved the time and cost of going back to the well. Simon's is the practitioner's answer.

Next: Ingeman Arbnor and Bjorn Bjerke, Methodology for Creating Business Knowledge (Sage, 1997). This book's purpose is to give the reader a foundation for handling questions about knowledge creation and for doing research efficiently and effectively in different situations and for various clients and employers. It is written as a basic university text in the theory of science and methodology in business.

The approach is very theoretical. It focuses on "four worlds of creating knowledge":

1. The Analytic Approach
2. The Systems Approach
3. The Actors Approach
4. The Relative View

As subsequent reading clarified for me,

1. The Analytic Approach is the positivist approach described above
2. The Systems Approach favors the development of explanatory models
3. The Actors Approach is more commonly known as "structuration theory," which derives from the work of Anthony Giddens, Central Problems in Social Theory (UCalP, 1979) and The Constitution of Society: Outline of the Theory of Structuration (Polity, 1984). It seems to take the position that social reality is determined by the subjective, but contextually shaped, responses of participants ("actors") to situations and events.
4. The Relative View is multi-methodological--a combination of approaches.

Interesting, I will admit, especially the sketches of major contributors to these ideas: Norbert Wiener, Charles West Churchman, George Mead, etc. But just try to relate these theories to the practical questions that crop up ever so often.

Most methodology courses, the authors observe, are organized around

• Formulating hypotheses
• Constructing questionnaires
• Controlling reliability and validity
• Sampling
• Statistical testing of hypotheses

Some cover historical descriptions and case studies. Few cover dialogic training [interviewing?] or exercises in language development.

As I turned to books on information systems research, it became clear to me that academics regard research in their area as a specialized kind of operations or organizational research, akin to sociological research. One article addressed "testing instruments" and never made clear whether the "instruments" were questionnaires or pieces of electronic hardware.

Maybe it's my prejudice, but when I looked over Michael E. Whitman and Amy B. Woszczynski, The Handbook of Information Systems Research (Idea Group, 2004), I found most useful the essays by academics on the edge of academe: people with substantial practitioner experience and/or working in university-sponsored business services organizations.

Joe McDonagh, "Investigating the Dynamics of IT-Enabled Change," looks at the study of organizational change via "action research," wherein the researcher is immersed in the organization under study. The author also focuses on "clinical inquiry," research done at the behest of and for the benefit of the client, but leading to discoveries of broader relevance. ("Action research" and "clinical inquiry" would seem to be at least compatible with "grounded theory.")

John C. Beachboard, "Rigor, Relevance and Research Paradigms," narrates his personal quest for answers about the nature of information systems research as he pursued first the MA in information resources management, then the PhD and an academic career. He observes that certain areas of interest are not studied academically because the study cannot be conducted with the scientific rigor required for publication in academic journals. Furthermore, "Much of the rigorous research was not relevant, much that was relevant was not rigorous."

M. S. Poole and G. DeSanctis, "Structuration Theory in Information Systems Research," note that the theory has been applied to

• Systems development
• Systems failure
• Virtual teams
• Online relationships
• Technology mediation
• Implementation of new information systems
• Organizational communication
• Organizational change.

This is an impressive list of significant areas for research, though all of them are on the "people" side of IT.

Regarding attitudes and trends, our authors state, variously:

The IS field has a deep-seated concern with analysis and design of structures for decision making and human-computer interaction. . . . No longer are IS researchers solely interested in structure as a property of technology. Our scope has extended to include structure as a property of workgroups, organizations, and other social entities that use IT. . . . Our concern is the interplay of people with technology--the structure of human-computer interaction, the structure of systems design and use, and the possibilities for somehow improving the human condition through applications of information technology to society.

Finally, in B. Kaplan, et al., Information Systems Research: Relevant Theory and Informed Practice (Kluwer, 2004), I found M. Jones, "Debatable Advice and Inconsistent Evidence: Methodology in Information Systems Research." Mr. Jones looked at a sample of the research literature and concluded that good research

• Follows the scientific method--but research practice in the natural sciences may bear little relation to "scientific method"
• Fulfills certain criteria (But which? The debate between positivist and interpretive approaches continues.)
• Is relevant--and relevance (a characteristic of the research product) is not necessarily incompatible with rigor (a characteristic of the research process)
• Is multi-methodological.

That said, our author shows that published research papers vary considerably in their account of such matters as methodology, sample size, data gathering, data analysis, etc. Evidently (my conclusion) it is more important to convince the journal board that the research was executed with rigor, less important to share the evidence of that rigor with anyone.

What can my colleague and I gather from this review of relevant literature? I would say that there are indeed certain issues and methods that should be included in any basic text on research in IT. Yet I would advise against emphasizing theory. Instead, I would include many examples that address typical questions, the choice of appropriate methods, and problems that may crop up. (I am reminded of my research methods course as an MA student in English. Our texts were primarily true stories of academic research: asking the question, finding the evidence, interpreting it, and publishing it.) Students heading toward an academic career will quickly adapt to the realities of academic publishing in the area.

Back for a reason

I've returned to blogging after a few years because I've committed to a reading program. This blog is devoted primarily to my reading and study in the areas of business analysis and information technology. I have no interest in publishing the details of my personal life or my political or religious opinions. I write because I need to consolidate my understanding of what I read. On occasion, I need to share my comments with friends and colleagues.

To quote from my earlier blog:

I use this blog for business communications and as an intellectual diary. I can report progress (and process) regarding a current project, and my clients or collaborators can have easy access to both the latest news and earlier reports. When I complete a report, I email the link, as an invitation to read, comment, and take action.

Why do I publish business reports in this public forum? I think they may provide information that may be useful to others doing the same thing. It's the old professor in me.

As intellectual diary, the blog is mostly book reviews and related reasoning: again, process and not just product. Note, though: I don't report on all my reading, but only that which seems related to the basic themes: creativity, communication, dialogue, access to information. Again, I publish because someone may find the information useful.

Earlier blog entries may be found at www.hal-pc.org/~mheumann/blogger.html and www.hal-pc.org/~mheumann/early_blogger.html. Topics include

• Art criticism
• Music criticism
  
• Social criticism
• Revamping an old PC
• Web-based communications and groupware
  
• Serving as foreman of a jury
• The Riyadh residential compound bombings
• Business consulting jobs
• Internet data mining and analysis
• Notes and comments on reading
• Creating a CD-ROM archive
• Treatment of mental illness and use of drugs
• Co-authoring a data modeling book
• How to analyze a poem
  
• Access to information in the humanities