How many times, when you were a kid, did a thunderstorm cause an immediate mix of emotions: the fear of the continuous boom of the thunder as the clouds rolled in, the calming sensation of the smell of freshwater in the breeze and wind, and the awe and wonder as the lighting streaked through the sky. For many of us, the mention of the words “Data Governance” can have the same effect. The awe and wonder that data and the information it provides could be organized and effectively managed, the calming sensation of having good quality data that is easy to reference and use, and the fear of trying to run a project in an attempt to implement data governance.
Yet with a recent project that my company completed and the additional enhancements that are underway, the right balance of cloud-based technology (in this case Informatica IDQ and PowerCenter hosted on AWS) and standard data governance methodologies provide a long-term data forecast that is clear and bright.
In order to accomplish a good long-term outlook for data governance, our project focused on five key areas:
- Data Traceability, Lineage, and Correctness
- Business users now have the capability to monitor the data from input to output and identify potential areas of concern.
- Business user can now remediate the data within a user interface that is connected to the cloud, as well as be able to export for additional analysis.
- If remediation is needed, the business user can follow a standard process to export the data to a spreadsheet, apply the needed corrections, and then reload the updated data.
- Exception handling processes, that can monitor lineage, correctness, and completeness, can be executed both with manual workflow management by the business team as well as with automated workflow processing.
- Additional features provided via the exception handling workflows are:
- Improved filter capabilities including Find and Replace option to quickly trace the data and determine the point of failure.
- Workflow tracking and progress metrics (e.g. number of exceptions in task, Status of accepted, rejected, Reprocessed).
- Scalability,
Flexibility and Tailored Services
- The solution now allows for rapid growth in the number of data quality (DQ) rules and the modifications that can be needed.
- Business users, via the IDQ user interface, can quickly activate and de-active business rules, based on the data sets being processed. This provides increased flexibility in reacting to data situations that require immediate attention and/or a short turnaround time.
- Informatica 10.1 on AWS is also less restrictive in reducing the time needed for user updates and deployments.
- Scalability is improved and streamlined due to the ease of maintenance in an AWS environment (i.e. processing capability, data sizing, etc.)
- Business
Empowerment and Wider Job Diversity
- The long-term goal for the IT support and service lines is to shift the administrative and solution delivery role such as design, operation, maintenance and scheduling roles to the business which will promote wider job diversity and growth of technical skills within the business.
- The IT support and service teams are now in a position to start focusing on technology and infrastructure configuration and enablement and, over time, transition the day-to-day execution of the Informatica solution to the business.
- Data Quality Profiling and Scorecard
Reporting Requirements
- Informatica provides a variety of reporting capabilities in profiling and score-carding that can be viewed via Informatica’s web-based user interface and the data can be exported / accessed via the cloud environment by 3rd party visualization tools for further drill down analysis.
- Informatica also provides capabilities to generate raw data in the desired template in a file for users for reporting which the business users plan to utilize in their long-term run and maintain processes.
- Ease of Use and Supportability
- Informatica provides enhanced product integration between the Developer and Analyst toolsets which eases the navigation and interaction within the development platform between the user interfaces.
- The business users have the ability to monitor, update, and execute their own set of business rules and logic, effectively managing the execution on daily basis; however, the solution preserves the safeguards and security for the IT service teams to properly maintain the environment and infrastructure.
Just as the weather can be, at times, unpredictable, aspects of data governance (data monitoring, cleansing, loaded, etc.) can also go awry. The key to effectively weathering the storm, as any good weatherman would say, is preparation, having the right tools, and sticking to the plan. By developing a data governance approach (preparation), providing a solution such as Informatica (the right tool), and having the end-users utilize the solution as a part of their daily operational process (sticking to the plan), organizations today can be ready to effectively ride out data storms that will come and know that clear skies are just over the horizon.
