Evaluating Decision Support System Projects

Why do we determine whether a computerized decision support system application is worth considering or not?

Why is there a need to predict the actual impact of a proposed DSS?

In short, the question is - why is it necessary to evaluate a decision support system project?

The answer is simple.

It’s done to assess the scope and benefits of a DSS. Evaluation is a systematic approach to identify whether its implementation will help resolve problems for which it’s being developed.

The primary purpose of evaluation is to reflect on to its usage and scope, along with identify the changes/upgrades it will need in future.

Evaluation of decision support system projects is a systematic, rigorous and painstaking application of scientific, statistical and mathematical methods to assess their design, implementation, outcome and improvement. Having said this, it’s a resource-intensive process that needs a sizeable budget, time, labor, human intelligence and unswerving dedication.

Evaluating a DSS project is important because an organization spends a lot of time and money in developing and implementing it. It must fulfill the expectations of users and aid in decision making.

Assessing its scalability is also important as technology is changing very frequently. It must be scalable and able to integrate new tools and web technologies whenever need arises.

When we evaluate a decision support system project, we must ask questions to:

• Understand evaluation techniques

• Examine technological risks

• Consider cross-cultural issues

• Assess return on investment

• Identify its superiority over previous DSS

• Determine payback period

Evaluating a DSS project may be difficult yet important. In this article, we focus on their evaluation process and issues and ethics that you may come across while evaluating them.

Decision Support System Project Evaluation Process

Evaluating DSS projects is an ongoing process, given their large scale and constant requirement in decision making. Evaluation process should be in proportion to the size, scope, complexity and cost of a proposed DSS.

Generally evaluation process is not much time consuming, if it’s not carried for a web-based project. Therefore, it’s important for managers to understand the quantity of information that will be required to evaluate a DSS. They must also mull over the type of evaluation that needs to be carried to assess a particular DSS.

When they consider evaluating a DSS, this very idea gives rise to a lot many related questions, such as:

• When should one evaluate a DSS?

• Do in-house capabilities need to be assessed while evaluating a DSS?

• Who will evaluate the DSS?

• How many resources will be needed to evaluate a DSS?

• Where will you gather the resources for evaluation?

• What will be the roles of members of the evaluating team?

• Will a feasibility analysis be required?

Asking these questions is important for a thorough evaluation of a DSS project. We will find answers to these questions along the way when we study the steps involved in a DSS project evaluation process.

DSS project evaluation is a multi-stage process and its scale determines the activities/steps to be taken to evaluate a project. One shouldn’t think that a project will work equally well years after as it does in present. Nor should one continue with a bad project just because they’ve invested time and money in building and implementing it.

DSS Evaluation Process

1. Idea Nurturing: DSS project evaluation begins with nurturing the idea. Even a small-scale DSS project idea should be evaluated. The moment the idea is in full bloom, it must be taken to the next stage, which is feasibility analysis. This gives a clear understanding about a proposed DSS.

2. Feasibility Analysis: The second step in the process is to conduct a SWOT (Strengths, Weaknesses, Opportunities and Threats) analysis of a proposed DSS project.

   A well-designed feasibility study provides a background of the project, a description of what it will do, details regarding research, technology, time, effort and money required for developing a project and legal, political, geographical ad cultural obligations to be fulfilled. Web-based DSS projects must be rigorously analyzed.

3. Scheduled Milestones: Managers should set some kind of milestones to check the DSS development on periodic basis. After each major milestone is achieved, they must stop and evaluate it. Before moving further, it will be ideal to fix any potential problems.

4. Evaluation Prior to Implementation: A DSS project will fail severely, if it isn’t carefully evaluated. It’s ideal to delay its implementation instead of rushing up with it without evaluating it.

5. Follow-up Evaluation: Note the feedback from initial user group as their experiences play a significant role in following up the evaluation or measuring the success of the evaluation. This step must be taken before introducing the system to the masses.

Evaluating DSS is a collaborative process. Digital analysts, managers, developers and decisions makers must come together to evaluate a DSS.

The number of resources required to DSS depends on latter’s size and complexity. Besides, it always pays up to evaluate in-house capabilities to evaluate a project. This will actually enhance the expertise of employees in using technological and other evaluation tools.

DSS Evaluation Tools and Techniques

While a number of tools and techniques are used to evaluate decision support systems, it’s important to determine which tool works best for your project. Let’s now learn about different DSS evaluation tools and techniques.

1. Cost-Benefit Analysis: Grounded in finance, cost-benefit analysis is a tool to determine whether a DSS is a sound investment or not. This tool is used to compare the total cost of development and implementation against total benefits that a system is expected to produce.

   If cost outweighs benefits, the idea is either further nurtured or dropped. Cost-benefit analysis tool takes into account hardware and software cost, personnel cost, process change cost, vendor/consultant cost, user training cost and infrastructure cost.

   Cost-Benefit Analysis Process

   Both costs and benefits can be tangible and intangible, hidden and explicit. Cost-benefit analysis only weighs the tangible aspects of a DSS project. Therefore, relying on this technique alone won’t be a great idea. Another tool must be deployed to approximate intangible aspects along with cost-benefit analysis.

2. Cost-Effectiveness Analysis: A simplified version of cost-benefit analysis, this method assumes that all of the alternatives either incur same costs or produce same benefits. This works when only costs or benefits are to be evaluated.

3. Incremental Value Analysis: This tool studies alternatives, fuels new ideas and raises what-if questions. The process emphasizes on the value offered by a proposed DSS rather than the cost incurred on it. The analysis is a multi-stage process with steps:

   • Establishing benefits that a DSS must achieve to be valuable;

   • Establishing the maximum cost that a company is willing to spend on building a DSS;

   • Building and assessing prototype Version 0;

   • Establishing cost and determining benefit verge for version 1

   • Developing version 1 and taking it to version N after monitoring costs and benefits

4. Qualitative Benefits Scenario Approach: As the name suggests, this method focuses more on quality of a DSS and its capabilities. The idea behind adopting this approach is to imagine possible future and assess and prepare for it.

   The aim is to determine whether or not a proposed DSS work will work equally well in future scenarios. Qualitative Benefits Scenario is a rigorous, long and a systematic process, with steps including:

   • Identification of trends and uncertainties

   • Building scenario themes

   • Conducting qualitative research

   • Developing decision scenarios

   • Visualizing the implementation of DSS

   • Explaining the use of a DSS

   • Discussing benefits resulting from a DSS

   • Assessing plausibility and consistency

   • Discussing risks and uncertainties

   • Estimating the minimum and maximum budget and development schedule

5. Research and Development Options Approach: The research and development options approach helps determine the cost of developing and maintaining flexibility to build a future DSS. It’s a three-step process:

   • Identifying the options to create future DSS opportunities in a given investment

   • Evaluating the circumstances and environment in which a company may want to invest more in a proposed DSS

   • Determining how much the company is willing to pay now to create future flexibility and opportunity

6. Scoring Approach: Scoring approach, when evaluating a proposed DSS, separates business and technical validation and considers those intangible benefits of a DSS that were not considered credible by DSS analysts who focused only on its financial aspect. Points are assigned to each criterion/benefit after reflecting on how well it satisfies a given factor.

   • Business validation involves assessment of strategic alignment, management information support, competitive advantage and organizational risk

   • Technical justification involves examining technical uncertainty, strategic systems architecture and system infrastructure risk

DSS Project Evaluation Criteria and Risk Factors

A DSS project is evaluated against four major criteria. What are those? Let’s learn about these here:

• Economic Criterion: This is one of the most important factors to be considered while evaluating any kind of project. It’s in fact, a measure of cost-effectiveness of a project.

• Operational Criterion: It is a measure of how well a proposed DSS will work in an organization and how its users feel about it.

• Schedule: It’s a measure of whether a proposed DSS can be developed and implemented in a reasonable time period.

• Technical Criteria: It’s a measure of technical viability and scalability as well as the practicability of a proposed DSS.

DSS Project Risk Factors

As the saying goes – everything has certain risk factors associated with it. A DSS project stands no exception to the rule. It is also associated with a number of risk factors. And risk increases with the ambiguity and disorderliness of objectives. The most common risk factors associated with a DSS project are:

• Narrow scope

• Low structure

• Inexperience of developers

• Unclear objectives

• Size of database

• Short-horizon cost saving

• Project failure because of the absence of analysis and evaluation

Although DSS project risk factors cannot be completely eliminated but they can certainly be subsided or reduced to a considerable extent. This can be done by assessing and evaluating various aspects of a project at each stage of its development.

International and Cultural Issues

As businesses nowadays operate in a global marketplace, decision support systems must eliminate issues and obstacles pertaining to its use in different countries. These issues could be:

• Culture: Some cultures support open communication while some support limited communication. There may be different meanings associated with a word. What one community finds normal may be unsuitable for the other. Managers must ensure that a group DSS takes care of such issues.

• Accounting and Currency Issues: Each country has a currency of its own. A group DSS used in different countries must support multiple currencies and specific accounting procedures and regulations.

• Impersonal Communication: Since much of the communication takes place electronically, there is a lack of personal communication. A communication-driven model can help resolve this issue and foster personal relationships between DSS users from different locations through online face-to-face meetings.

• Language Incompatibility: English is considered ‘the language’ in the business world. But it’s not necessary that everyone across the globe understands it equally well. For a multi-lingual staff, a DSS must support multiple languages and translation tool.

• Time Zone Differences: Different time-zones make it harder for people to conduct real meetings even online. A DSS must support tools to store and display information offered by a user, so that it can be seen and understood by the other users working in different time zones.

Ethics and Privacy Issues

In addition to international and cultural issues, decision support systems are exposed to ethics and privacy problems. A DSS project fails dramatically, if it fails to address these concerns. As a developer or a manager, you may think that a computerized system is ethically neutral and therefore, it shouldn’t raise any ethical issues. Wrong!

You’re ignoring the importance of values and principles that go into decision making. When knowledge or model driven DSS are used, each choice is known to have made an ethical impact. Certain people may resort to a particular technique while some will not because there conscious or their principles don’t allow them to use it.

Therefore, while constructing a decision support system, all such issues must be taken care of, in order to ensure its smooth implementation and use. In addition, privacy concerns must be considered during the evaluation of a project.

The users expect a system to keep their personal, behavioral and habitual information private. The exact extent of privacy though cannot be defined but there should be an arrangement where DSS users can secure their private information.

Final Thoughts: Decision support systems are not a new technology. Rather they are being used more frequently. However, it’s really important to evaluate the scope of a project before taking it off the ground. This gives a clear idea of whether its implementation will be beneficial for a company or not.