发帖
雷达卡
要做数据分析，首先解决这两类数据质量问题！
Defining your data quality problems
为了能够系统化地、高效地解决出现的任何问题，我们必须学会将这些问题分而治之。毕竟，知己知彼方是解决问题的首重至要。由此，我们才会发现解决之道就在其中。而对于提高数据质量同样适用：每一个解决问题的方法都有不同的阶段与角度。
To ackle any problem in a systematic and effective way, you must be able to break it down into parts. After all, understanding the problem is the first step to finding the solution. From there, you can develop a strategic battle plan. With data quality, the same applies: every initiative features many stages and many different angles of attack.
点击进入『21世纪大数据人才汇聚领域』
点击进入『21世纪数据分析精英聚集地』
点击进入『21世纪大数据高薪就业领地』
当一个数据质量改进程序在启动时，仅知道数据库中有多少错误计算或重复录入是远远不够的。不止于此，我们还需要知道不同类型的错误在收集的资源中是如何分配的。
When starting a data quality improvement program, it’s not enough to count the amount of records that are incorrect, or duplicated, in your database. Quantity only goes so far. You also need to know what kind of errors exist to allocate the correct resource.
据 Jim barker 一篇很有意思的博客所述，数据的质量被分解成两种不同类型。而在本文中，我会带领大家仔细区分这些“类型”有何不同，并且如何利用这些“类型”在开发预算中确保我们的优势资源放在何处。
In this interesting blog by Jim Barker, the different types of data quality are broken down into two parts. In this article, we’ll look closely at defining these ‘types’, and how we can use this to our advantage when developing a budget.
数据类型
Types of Data
被誉为“数据博士”的Jim barker，借用了一个简单的医学概念来定义数据质量问题。 在他的博客中介绍了如何将这两种“类型”组合在一起，并且成功激发了那些一直纠结于找到在数据库中拉低数据质量的幺蛾子的数据分析师们的兴趣。
I型数据质量问题我们可以使用自动化工具检测到。II型数据质量问题就非常隐秘了。大家都知道它是存在的，但它看不见摸不着，更处理不了，因为它需要放在特殊情境才能被检测到。
Jim Barker – known as ‘Dr Data’ to some – has borrowed a simple medical concept to define data quality problems. His blog explains just how these two types fit together, and will be of interest to anyone who has struggled to find the data quality gremlins in their machine.
On the one hand, there’s the Type I data quality problem: things we can detect using automated tools. On the other hand, Type II is more enigmatic. You know the data quality problem is there, but it’s more difficult to detect and deal with, because it needs to be contextualised to be detected.
它们之间的区别简而言之可归纳为如下几点：
The key differences can be simply and quickly defined:
I型数据质量问题首先需要“知其然”才能来检测数据的完整性、一致性、唯一性和有效性。这些属性靠数据质量软件甚至手动很好地找到。你不需要有很多的背景知识，或者数据分析经验。只要按照4个属性验证它的存在，就可以判定它错误的。例如，如果我们在性别领域插入一个3，我们就可以判定它到底是不是一个有效值。
Type I data quality problems require “know what” to identify: completeness, consistency, uniqueness and validity. These attributes can be picked up using data quality software, or even manually. You don’t need to have a lot of background knowledge, or a track record working with that data. It’s there, it’s wrong and you can track it down. For example, if we insert a 3 into a gender field, we can be sure that it is not a valid entry.
II型数据质量问题需要“知其所以然”来检测时效性、一致性和准确性属性。需要研究能力、洞察力和经验，而不是简简单单就可以找得出来的。这些数据集经常从表面上看起来没有问题。但幺蛾子往往存在于细节中，需要时间去发现。Jim举的例子就是一份退休人员的雇佣记录。如果我们不知道他们早已退休的话，是看不出来这个数据是错的。
Type II data quality problems require “know how” for detection of timeliness, congruence and accuracy attributes. They require research, insight and experience and are not as simple or straightforward to detect. These datasets may appear free of problems, at least on the surface. The devil is in the detail, and it takes time to correct. Jim’s example is an employee record for someone who has retired. Without knowing the date of retirement, their data would otherwise appear to be correct.
所以，解决这些数据质量问题的关键就是需要一个复杂的、战略化的方法，而非孤立的、片面的来看问题。一旦数据质量不好，我们就需要寻求自动化与人工的方式才能解决这个问题了，真可谓是“屋漏偏逢连夜雨”啊。
The key takeaway is that data quality problems require a complex, strategic approach that is not uniform across a database. Once we break the data down, we start to see that it requires human and automated intervention – a dual attack.
成本调整
Cost to Fix
所以，我们如何解决I型和II型数据质量问题呢？处理它们所花费的费用是可比的，还是完全不同的？
要记住重要的一点是，I型数据的验证问题可以在逻辑上定义，这意味着我们可以靠编写软件来查找并显示它。软件自动修复的速度快、成本低，甚至配合手动审查就可以完成。考虑到I型数据质量问题实际上是作为表格内字段型的验证，一旦解决了表格字段的问题，I型数据质量问题实际上也就解决了。
So, how do we deal with Type I and Type II data quality problems? Are the costs comparable, or are they different beasts entirely?
The important thing to remember is that a Type I data validation or verification problem can be logically defined, and that means we can write software to find it and display it. Automated fixes are fast, inexpensive and can be completed with only occasional manual review. Think of Type I data quality problems as form field validation. Once valid, the problem disappears.
根据我们以往的经验：I型数据基本涵盖了80%的数据质量问题，但消耗了我们20%的经费成本。
We could estimate that Type I data presents 80 per cent of our data quality problems, yet consumes 20 per cent of our budget.
第二类数据问题往往需要多方的输入，以便发现、标记和根除。虽然我们客户关系管理系统中的每个人都有购买日期，但购买日期可能不正确，或者与发票或发货清单不符。只有专家才能通过仔细核查其内容来解决问题并手动改进客户关系管理系统。
Type II data needs the input of multiple parties so that it can be discovered, flagged up and eradicated. While every person in our CRM may have a date of purchase, that purchase date may be incorrect or not tally with an invoice or shipping manifest. Only specialists will be able to seed out problems and manually improve the CRM by carefully verifying its contents.
通常情况下，企业很难做到资源的合理分配，原因有二，特别是企业处于快速增长阶段；或者处于人才流失的时候。你别看这些II类问题较少，可能仅占数据问题剩余的20%，但它们很有可能需要消耗超过80%的成本预算。所以，如果当企业处于人才大量流失，却又对此无能为力的时候。你会发现第二类数据问题更难处理，因为人工解决的途径已不复存在了。
Often, businesses find it difficult to allocate the necessary resource – particularly if they have grown rapidly, or have high employee churn. While these Type II problems are fewer – perhaps the remaining 20 per cent of the database – they could require 80 per cent of our data quality budget, or more. If you continually lose staff who have that knowledge, and you fail to retain any of it over time, you will find Type II data much more difficult to deal with because the human detection element is lost.
提高精确程度
Improving Accuracy
为了提高数据的准确性，我们必须将I型和II型数据问题作为单独的，但同时存在的问题进行研究。I类型数据质量的挑战可以呈现快速获胜，但第II类问题提出了一个挑战，必须依靠人类的专业知识才可以解决。
In order to improve data accuracy, we must work on Type I and Type II data as separate, but conjoined, problems. Fixing Type I data quality challenges can present quick wins, but Type II presents a challenge that human expertise can solve.
随着时间的推移，数据库会超过使用期限。为保其时效性，这需要持续不断的努力。数据可以在数据库中进行清洗，或在使用阶段进行清理，但由于如导入/导出、损坏、手动编辑、人为导致错误等多种原因，仍然要注意I型错误的发生。第II类数据问题在这阶段自然而然地发生，因为就算数据经过验证和审查之后看起来正确，但对于现在来说仍有可能是不正确的，因为此时已非彼时，数据的使用环境改变了。
Over time, a database will always drift out of date, and this requires on-going and sustained effort. Data can be cleansed in situ, or validated at the point of entry, but Type I errors will still occur for a number of reasons; import/ export, corruption, manual edits, human error. Type II data problems will occur naturally, of their own accord; data that validates and looks correct may now be incorrect, simply because someone’s circumstances have changed.
确保数据的完整
Ensuring Data Integrity
数据的完整会有助于我们观察整个事物的全貌并推动其对事物的决策。正如我们前面所说，发现I型数据质量问题是比较简单、廉价和快速的。但如果企业的工作业务还没有采用某种数据质量软件来解决I型数据质量问题的话，那现在也应该着手考虑了，因为这样才可能避免将来出现的资源浪费、损害品牌效应和来自大众的误解。
Data informs business decisions and helps us get a clear picture of the world. Detecting Type I data quality problems is simple, inexpensive and quick. If your business has not yet adopted some kind of data quality software, there’s no doubt that it should be implemented to avoid waste, brand damage and inaccuracy.
而对于第II类数据问题，关键是要理解它为什么会发生，并采取措施以防止它的发生。从日常工作中，处事的变通以及员工疏忽常导致数据的质量不佳。随着时间的推移，资源分配失当也会增加II型数据问题的增加。而改善它的费用也会成倍增加，因为你需要具备专家的眼光方能在茫茫的数据中找到它的存在。
As for Type II, the key is to understand that it exists and to implement new processes to prevent it from occurring. Workarounds and employee diversions from business processes will drag the data down. A failure to allocate subject matter experts could increase the amount of Type II over time. And as the proportion increase, so does the price of fixing it, because you need expert eyes on the data to weed it out. See the 1:10:100 Rule article.
其实，发现并解决这两类问题在当下已不是不可能的事了。会变得越来越容易。很多数据质量供应商们也在不断寻找新的方法，相信在不远的将来，得到高质量的数据会变得越来轻松，越来越简单。
Detecting and eradicating both types of problem is not impossible. One is easier than the other. Data quality vendors are continually looking at new ways to make high quality data simpler to achieve.
原作者：Martin P Doyle
提升卡
置顶卡
沉默卡
变色卡
抢沙发
千斤顶
显身卡
京公网安备 11010802022788号
