Abstract<\/p>\n

Have you ever been working on an unsupervised task and wondered, \u201cHow you I validate my algorithm at scale?\u201d<\/p>\n

In unsupervised learning, in contrast to supervised learning, our validation set has to be manually created and checked by us, i.e. we will have to go through the classifications ourselves and measure the classification accuracy or some other scores. The problem with manual classification is the time, effort, and work that is required for classifications, but this is the easy part of the problem.<\/p>\n

Let\u2019s assume that we developed an algorithm and tested it very well while manually passing on all the classifications, what about future changes to that algorithm? After every change we should check the classifications manually ourselves again. While the data classified might change with time, it might also grow to huge scales with the evolution of our product, and the growth of our customers, then our manual classification problem would of course be much more difficult. <\/p>\n

Have you started to worry about your production algorithms already? Well, you shouldn\u2019t! <\/p>\n

After reading this, you will be familiar with our proposed method to validate your algorithm score easily, adaptively, and effectively against any change in the data or the model.<\/p>\n

So let’s start detailing it from the beginning.
\nWhy is it needed?<\/p>\n

Algorithm continuous modifications always happen. For example, we are having:<\/p>\n

\tRuntime optimizations
\n\tModel improvements
\n\tBug fixes
\n\tVersion upgrades<\/p>\n

How are we dealing with those modifications? We usually use QA tests to make sure the system keeps working. At the same time, the best among us might even develop some regression tests to make sure, for several constant scenarios, that the classifications would not be changed
\nWhat about data integrity?<\/p>\n

But what about the real classifications on prod? Who verifies their change? We need to make sure that we won\u2019t have any disasters on prod when deploying our new changes in the algorithm.<\/p>\n

For that, we have two optional solutions:<\/p>\n

\tNaive solution – pass through all the classifications on prod (which is of course not possible)<\/p>\n

\tPractical solution – use samples of each customer data on prod – using the margin of error equation.<\/p>\n

Margin of error <\/p>\n

To demonstrate, we are going to take a constant sample from each customer\u2019s data, which would represent the real distribution of the data with minimal deviation, which we will do using the Margin of Error equation, sometimes known from election surveys, where the surveys are sometimes based on some equation derived from the Margin of Error equation.<\/p>\n

So, how does it work?<\/p>\n

We can use the first equation used for calculating the margin of error, to extract the needed sample size desired.<\/p>\n

We would like to have a maximum margin of error of 5%, while we should use a constant value of Z = 1.96 if we want the confidence of 95% (might be changed if we would like to have another confidence level)<\/p>\n

The extraction of the required sample size is demonstrated in the following equation:<\/p>\n

While this equation is an expansion of the equation above, it might be used when we have the full data size, to be more precise. Otherwise, we\u2019ll be left only with the numerator of that equation – which is also fine if we don\u2019t have the full data size.<\/p>\n

This is a code block demonstrating the implementation of this equation in Python:<\/p>\n

We can now freeze those samples, which we call a \u201cgolden dataset,\u201d and use them as a supervised dataset that will be used by us in the future when making modifications, and serves us as a data integrity validator on real data from prod.<\/p>\n

We should mention that because optional changes on prod data might happen with time, we encourage you to update this golden dataset from time to time.
\nThe flow of work for end-to-end data integrity:<\/p>\n

\tManual classification to create a golden dataset
\n\tMaintaining a constant baseline of prod classifications
\n\tDeveloping a suite of score comparison tests
\n\tIntegrating quality check into CI-process of the algorithm<\/p>\n

So, how will it all work together? You can see that in the following GIF:<\/p>\n

We may now push any change to our algorithm code, and remain protected, thanks to our data integrity shield! <\/p>\n

For further questions about data integrity checks, or data science in general, don\u2019t hesitate to reach out to me at emeyuhas@zscaler.com.\u00a0\u00a0<\/p>\n

\u200b<\/p>\n

Margin of error\u00a0<\/strong><\/h3>\n

To demonstrate, we are going to take a constant sample from each customer\u2019s data, which would represent the real distribution of the data with minimal deviation, which we will do using the Margin of Error equation, sometimes known from election surveys, where the surveys are sometimes based on some equation derived from the Margin of Error equation.<\/p>\n

\n<\/p>

So, how does it work?<\/p>\n

We can use the first equation used for calculating the margin of error, to extract the needed sample size desired.<\/p>\n

We would like to have a maximum margin of error of 5%, while we should use a constant value of Z = 1.96 if we want the confidence of 95% (might be changed if we would like to have another confidence level)<\/p>\n

\n\u00a0<\/p>\n

The extraction of the required sample size is demonstrated in the following equation:<\/p>\n

\n<\/p>

While this equation is an expansion of the equation above, it might be used when we have the full data size, to be more precise. Otherwise, we\u2019ll be left only with the numerator of that equation – which is also fine if we don\u2019t have the full data size.<\/p>\n

This is a code block demonstrating the implementation of this equation in Python:<\/p>\n

<\/strong><\/p>\n

We can now freeze those samples, which we call a \u201cgolden dataset,\u201d and use them as a supervised dataset that will be used by us in the future when making modifications, and serves us as a data integrity validator on real data from prod.<\/p>\n

We should mention that because optional changes on prod data might happen with time, we encourage you to update this golden dataset from time to time.<\/p>\n

\u00a0Abstract<\/p>\n

Have you ever been working on an unsupervised task and wondered, \u201cHow you I validate my algorithm at scale?\u201d<\/p>\n

In unsupervised learning, in contrast to supervised learning, our validation set has to be manually created and checked by us, i.e. we will have to go through the classifications ourselves and measure the classification accuracy or some other scores. The problem with manual classification is the time, effort, and work that is required for classifications, but this is the easy part of the problem.<\/p>\n

Let\u2019s assume that we developed an algorithm and tested it very well while manually passing on all the classifications, what about future changes to that algorithm? After every change we should check the classifications manually ourselves again. While the data classified might change with time, it might also grow to huge scales with the evolution of our product, and the growth of our customers, then our manual classification problem would of course be much more difficult. <\/p>\n

Have you started to worry about your production algorithms already? Well, you shouldn\u2019t! <\/p>\n

After reading this, you will be familiar with our proposed method to validate your algorithm score easily, adaptively, and effectively against any change in the data or the model.<\/p>\n

So let’s start detailing it from the beginning.
\nWhy is it needed?<\/p>\n

Algorithm continuous modifications always happen. For example, we are having:<\/p>\n

\tRuntime optimizations
\n\tModel improvements
\n\tBug fixes
\n\tVersion upgrades<\/p>\n

How are we dealing with those modifications? We usually use QA tests to make sure the system keeps working. At the same time, the best among us might even develop some regression tests to make sure, for several constant scenarios, that the classifications would not be changed
\nWhat about data integrity?<\/p>\n

But what about the real classifications on prod? Who verifies their change? We need to make sure that we won\u2019t have any disasters on prod when deploying our new changes in the algorithm.<\/p>\n

For that, we have two optional solutions:<\/p>\n

\tNaive solution – pass through all the classifications on prod (which is of course not possible)<\/p>\n

\tPractical solution – use samples of each customer data on prod – using the margin of error equation.<\/p>\n

Margin of error <\/p>\n

To demonstrate, we are going to take a constant sample from each customer\u2019s data, which would represent the real distribution of the data with minimal deviation, which we will do using the Margin of Error equation, sometimes known from election surveys, where the surveys are sometimes based on some equation derived from the Margin of Error equation.<\/p>\n

So, how does it work?<\/p>\n

We can use the first equation used for calculating the margin of error, to extract the needed sample size desired.<\/p>\n

We would like to have a maximum margin of error of 5%, while we should use a constant value of Z = 1.96 if we want the confidence of 95% (might be changed if we would like to have another confidence level)<\/p>\n

The extraction of the required sample size is demonstrated in the following equation:<\/p>\n

While this equation is an expansion of the equation above, it might be used when we have the full data size, to be more precise. Otherwise, we\u2019ll be left only with the numerator of that equation – which is also fine if we don\u2019t have the full data size.<\/p>\n

This is a code block demonstrating the implementation of this equation in Python:<\/p>\n

We can now freeze those samples, which we call a \u201cgolden dataset,\u201d and use them as a supervised dataset that will be used by us in the future when making modifications, and serves us as a data integrity validator on real data from prod.<\/p>\n

We should mention that because optional changes on prod data might happen with time, we encourage you to update this golden dataset from time to time.
\nThe flow of work for end-to-end data integrity:<\/p>\n

\tManual classification to create a golden dataset
\n\tMaintaining a constant baseline of prod classifications
\n\tDeveloping a suite of score comparison tests
\n\tIntegrating quality check into CI-process of the algorithm<\/p>\n

So, how will it all work together? You can see that in the following GIF:<\/p>\n

We may now push any change to our algorithm code, and remain protected, thanks to our data integrity shield! <\/p>\n

For further questions about data integrity checks, or data science in general, don\u2019t hesitate to reach out to me at emeyuhas@zscaler.com.\u00a0<\/p>","protected":false},"excerpt":{"rendered":"

Abstract Have you ever been working on an unsupervised task […]<\/p>\n","protected":false},"author":0,"featured_media":2084,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"class_list":["post-2083","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-zenith-zscaler"],"yoast_head":"\n